twchennai / gocd-git-path-material-plugin Goto Github PK

Unable to connect to TFS Git repo:
Invalid URL format. Should match ^(?:git|ssh|https?|git@[\w-\.]+[\w]+\.[\w]+):(?:\/\/)?[\w\.@:\/~_-]+\.git(?:\/?|\#[\d\w\.\-_]+?)$
I think it's because URL format doesn't ends with "*.git"
Is it supported with this plugin?
Also need to authenticate with TFS token or domain user. Is it possible?

Every time a pipeline fails with 17.3 windows 64bit agent we get this error:

11:16:28.120 [go] Job Started: 2017-04-17 11:16:28 UTC

11:16:28.120 [go] Start to prepare myPipe/5/Package/1/CreateNuget on MVDC [C:\GA]
11:16:28.214 Clean working directory is set to true. Unable to clean working directory for agent: C:\GA\pipelines\myPipe, with error: Unable to delete file: pipelines\myPipe\local\.git\objects\pack\pack-7424f80058b68300dfd2f2c7e2b67fad474b1e8f.pack
11:16:28.261 [go] Job completed myPipe/5/Package/1/CreateNuget on MVDC [C:\GA]

We have to manually restart the GoAgent. This happens only with this plugin.

gocd-server: 20.3.0
git-path-material-plugin: 2.1.0-78
webhook: bitbucket

Hi, I am struggling to get the pipeline to be triggered by webhook event, currently I have 2 repositories setup 1 setup using standard material and the second one is using the git-path-material-plugin to listen for a specific path change.

When pushing a change the 1st pipeline starts as expected but the 2nd one does not start regardless of what change I make in the path it is supposed to listen.

GoCD server allows use of shallow clones on its Git materials on agents which can massively improve clone performance on larger repositories.

Looks like jgit runs in debug mode and it spits out a lot of debug log statements, which are forwarded to go-server.out.log.

There is another issue in the go server [https://github.com/gocd/gocd/issues/2811] and the log file go-server.out.log is not rotated.

It results in the log file taking the entire disk space.

While the rotation issue shall be handled in the go server, there is no reason for the jgit and the plugin to run in debug mode.

Is it possible to get the to and from revisions for a material when using this plugin? When I take a look at the envvar's, I only see the revisions for the materials using the standard set up from gocd.

I'd like to see the triggering revisions if possible:

[go] setting environment variable 'GO_REVISION_mat_name' to value 'abc123'
[go] setting environment variable 'GO_TO_REVISION_mat_name' to value 'abc123'
[go] setting environment variable 'GO_FROM_REVISION_mat_name' to value '123abc'

CONTEXT
I have two go-cd server installed in two different servers (GO1 in M1 and GO2 in M2) with gitpathmaterial installed manually and a git-server installed in another server (GIT in M3) under the same subnet of the M2.
Go-cd version is 19.8.0 and GitPathMaterial version is 1.2.3. I already know that is obsolete and we have planned to upgrade but actually we need to find a way to work out.

PROBLEM
The pipelines linked with gitpathmaterial in M1 don't start at the same time of the M2s. Reading the GitPathMaterial Log of M2 is constantly writing, as if it pings frequently. M1's Log instead it seems frozen but if I change some reference in material it write something, so it's working but it's not polling at the same way of M2s.
I found out that when I ping from M2 the git server the time response is below 1 ms (eg. time=0.478 ms). From M1, instead, the ping is time=19.7 ms. I don't know if it's related but there is a way to set a time of polling? or to force the polling so it starts immediatly the pipeline?

We are using this plugin, in combination with the gocd-kubernetes-based-secrets-plugin to specify the username and password to access a Git material over HTTPS with authentication.

When the server fails to clone/ make a git request, the error message exposes the username and password combination, leaking git credentials in the process, e.g.:

Modification check failed for material: [url=https://gitlab-instance/project.git, username={{SECRET:[deploy-token][username]}}, password={{SECRET:[deploy-token][token]}}, branch=main] Affected pipelines are pipeline-A.
The plugin sent a response that could not be understood by Go. Plugin returned with code '500' and the following response: '"latest-revision failed due to [RuntimeException: Exception (Process exited with an error: 128 (Exit value: 128)) Occurred: [git, clone, --branch=main, --no-checkout, https://username:exposed-password@gitlab-instance/project.git, /go/pipelines/mypipeline/my-material] - null], rootCause [ExecuteException: Process exited with an error: 128 (Exit value: 128)]"'

My company has recently moved from svn to git for source control so I have a legacy application that consists of multiple projects which all reside in the same git repository (not ideal I know). Now I have been attempting to use the git path material plugin to setup CI for specific components, however I am getting the bellow error message and due to receiving this error go is repeatedly trying to download the code into the flyweight directory. Anyone else encountering this? Running 20.2.0 on Windows.

(changed the uri path slightly and * out the user id)
[132@MessageListener for MaterialUpdateListener] MaterialDatabaseUpdater:127 - [Material Update] Modification check failed for material: [url=https://gitlab.com/projects/myproject.git, username=*****, path=\server\services\Notification, branch=master]
java.lang.RuntimeException: Unable to de-serialize json response. SCM revision cannot be empty
at com.thoughtworks.go.plugin.access.scm.JsonMessageHandler1_0.toSCMRevision(JsonMessageHandler1_0.java:312)
at com.thoughtworks.go.plugin.access.scm.JsonMessageHandler1_0.responseMessageForLatestRevision(JsonMessageHandler1_0.java:156)
at com.thoughtworks.go.plugin.access.scm.SCMExtension$5.onSuccess(SCMExtension.java:117)
at com.thoughtworks.go.plugin.access.scm.SCMExtension$5.onSuccess(SCMExtension.java:109)
at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:54)
at com.thoughtworks.go.plugin.access.scm.SCMExtension.getLatestRevision(SCMExtension.java:109)
at com.thoughtworks.go.server.service.materials.PluggableSCMMaterialPoller.latestModification(PluggableSCMMaterialPoller.java:60)
at com.thoughtworks.go.server.service.materials.PluggableSCMMaterialPoller.latestModification(PluggableSCMMaterialPoller.java:45)
at com.thoughtworks.go.server.service.MaterialService.latestModification(MaterialService.java:120)
at com.thoughtworks.go.server.materials.LegacyMaterialChecker.findLatestModification(LegacyMaterialChecker.java:50)
at com.thoughtworks.go.server.materials.ScmMaterialUpdater.insertLatestOrNewModifications(ScmMaterialUpdater.java:55)
at com.thoughtworks.go.server.materials.ScmMaterialUpdater.addNewMaterialWithModifications(ScmMaterialUpdater.java:70)
at com.thoughtworks.go.server.materials.PluggableSCMMaterialUpdater.addNewMaterialWithModifications(PluggableSCMMaterialUpdater.java:61)
at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.addNewMaterialWithModifications(MaterialDatabaseUpdater.java:178)
at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.initializeMaterialWithLatestRevision(MaterialDatabaseUpdater.java:136)
at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater$1.doInTransaction(MaterialDatabaseUpdater.java:95)
at com.thoughtworks.go.server.transaction.TransactionCallback.doWithExceptionHandling(TransactionCallback.java:23)
at com.thoughtworks.go.server.transaction.TransactionTemplate.lambda$executeWithExceptionHandling$2(TransactionTemplate.java:43)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
at com.thoughtworks.go.server.transaction.TransactionTemplate.executeWithExceptionHandling(TransactionTemplate.java:40)
at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.updateMaterial(MaterialDatabaseUpdater.java:92)
at com.thoughtworks.go.server.materials.MaterialUpdateListener.onMessage(MaterialUpdateListener.java:64)
at com.thoughtworks.go.server.materials.MaterialUpdateListener.onMessage(MaterialUpdateListener.java:32)
at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.runImpl(JMSMessageListenerAdapter.java:83)
at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.run(JMSMessageListenerAdapter.java:63)
at java.base/java.lang.Thread.run(Unknown Source)

Let me take a step back on this and ask if anyone has an example of how to properly configure a git path material?

Using 20.2.0

step to reproducue :-

1. name any material for git e.g. test1 and corresponding url.
2. delete the pipeline.
3. create the same name pipeline and use same material previously used.

Result :-
not able to save the add pipeline with this change giving below error :-

Cannot save SCM, found multiple SCMs called 'test1'. SCM names are case-insensitive and must be unique.

@nitink85

GoCD Server's main git material does --no-checkout clones server side when clone repositories in order to watch them for updates on materials. This support would be useful to make the Git Path a drop-in replacement for the off-the-shelf Git material; since this is a more performant approach server-side when building changelogs and checking for versions.

Hi,

When I set git password fetched using secrets (file-based-plugin), material fails.
I configured build stage of pipeline as pipeline-as-code as below

materials:
  git-myrepo:
        plugin_configuration:
          id: git-path
        options:
          url: https://github.com/myrepo.git
          branch: master
          username: user1
          path: src/**
          password: '{{SECRET:[github-credentials][password]}}'
          shallow_clone: true

Appreciate if you can reply to this.

Thanks
Rak

Hello, we have microservice monorepo and have noticed that plugin checks out latest git revision that only touched paths that plugin monitors. We have one material per service, and each material monitors on its service subpath. When some service is about to be built & deployed, plugin picks up latest commit that touched the service. This is problematic as we want build phase to always checkout latest revision from master instead, as there are indirect dependencies between microservices. Is there a way to enforce plugin to do this for build phase?

It would be very helpful to be able to skip a build based on the commit message, either matching for a regex or purely looking for a [ci skip] in the string.

We use a monorepo and update the version information of our packages in an automated commit, it would be great if we could skip that subsequent build.

Prior art:
https://docs.travis-ci.com/user/customizing-the-build/#skipping-a-build

Thank you!

Currently the integration with the git command line client doesn't have great error messages when the git command line exits for some reason when trying to clone/fetch/reset a target repo within an agent.

This is one distinct disadvantage of this plugin from the built-in Git material; it's a bit painful if the Git repo server is having issues; or when diagnosing credential errors (for example), that might only be present on agent instances, but not on the server (e.g connectivity issues; issues with git SSH credentials/keys).

The changes in #24 improved this a little bit by including the git command line run; however this could be improved further:

The plugin sent a response that could not be understood by Go. Plugin returned with code '500' and the following response: '"checkout failed due to [RuntimeException: Exception (Process exited with an error: 128 (Exit value: 128)) Occurred: [git, fetch, origin, --prune, --recurse-submodules=no] - /go/pipelines/my-pipeline/my-material], rootCause [ExecuteException: Process exited with an error: 128 (Exit value: 128)]"'

Exit code 128 just means "git did not exit cleanly".

This probably requires some changes upstream at git-cmd however the basic mechanisms to get this data are already there in terms of being able to collect the streams off stdout and stderr. Just need to find a more standard way to return this back as a nicer error message to the GoCD Server.

Workaround
Set GIT_TRACE=1 env var (and sometimes GIT_SSH=1...) on the agent... but this is very noisy.

This plugin is only working with https:// style git URLs, and fails to connect with git@ style repository URLs.

I have GOCD 19.8.0 with GitPathMaterial Plugin 1.2.3. I cannot upgrade because all the pipeline must run with at MAX java 8.
I have multiple pipeline stuck to an older commit.

Based on log analysis, the flyweight-folder point to the same UUID path and the revision did not change.

There is a way to purge the latest-revision-since or clean up the history to make the pipelines start as a first time poll to git repository?

thanks

I'm trying to use pipeline parameters for defining the material with git path material plugin. Here's how my definition looks for a material:

my_pipeline:
  group: build
  parameters:
    git-url: [email protected]:test/test-gocd.git
    project-path: sub_pj1
 materials:
   dev-material:
     plugin_configuration:
       id: git-path
     options:
       url: "#{git-url}"
       path: "#{project-path}"
       branch: develop
     destination: my_project

I see a 128 error code when GoCD is trying to parse this pipeline config.

The default git material that gocd has allows this type of material declaration but not the git path material.
This is particularly useful for projects that have huge number of microservices in which we can use yaml aliases for the materials and just define the overriding parameters.

similar to gocd-build-github-pull-requests support pull request for the specified branch and path

I've been trying to understand how to achieve ignoring certain paths. As of now, only Wildcard whitelists are supported according to the README. Is this the case, or am I missing something here?

Is the blacklist functionality planned for the future?

I've tested out this plugin and it's working great, except my source repo is split up according to Pants preferred way. Meaning that my source code lives in src/<lang>/company/<project>/etc and tests in tests/<lang>/company/<project>/etc.

Does this plugin support multiple paths to be specified?
I tried looking through the source, but I couldn't figure it out through GitHub.
If it doesn't, I wouldn't mind having a stab at adding that feature, but I would love any pointers for how I should approach it. :)