Support for ignore list / blacklist about gocd-git-path-material-plugin HOT 10 CLOSED

Currently by design, it is/was intended/validated as a whitelist. There is no specific plan for blacklist.

One way to think about what is possible is "can you express this with a call to git log with particular pathspec arguments?", as that is essentially what the plugin does - relies on the ability to give 0..* paths to git commands in order to filter the output.

However, it might work out of the box, if you specify the paths using git pathspec exclude format like

git log 'path/to/stuff' ':^path/to/stuff/excludeme' or
git log 'path/to/stuff' ':!path/to/stuff/excludeme'

In which case you would express this in git path as monitored paths path/to/stuff,:^path/to/stuff/excludeme.

Not tested this though; would be interesting to see if it works.

from gocd-git-path-material-plugin.

@chadlwilson

We tried the approach you suggested and it works. We used the following pathspec:

:(exclude)path/to/exclude/1,:(exclude)path/to/exclude/2

Thank you for resolving this!

from gocd-git-path-material-plugin.

However, we have a related issue. After implementing includes/ignores using git-pathspec for the same git repository, we see the following error:

Cannot save SCM, found multiple SCMs called 'my-repo'. SCM names are case-insensitive and must be unique.

However the pipelines seem to be running fine, but they show up as two different materials in the VSM:

And the fan-in does not seem to be working as well. A commit that triggers both the materials still individually trigger a downstream pipeline.

from gocd-git-path-material-plugin.

If you have different git-path path: rules across those two materials, you need to name them uniquely - that is what GoCD is telling you with that error.

I think you're also possibly misunderstanding how fan-in can/should work - you can only fan in identical materials that use the same triggering mechanism, e.g

 /-- P1 --\
M          P3
 \-- P2 --/

commit to material M will trigger both P1 and P2. Fan-in will be at P3 (only if P1 and P2 have been run with the same revision of the upstream material will P3 trigger). If the pipeline sees them as two logically different materials as with your screenshot above, that wouldn't be fan-in - just independent triggering materials.

from gocd-git-path-material-plugin.

We do have unique names for the materials.

The only difference between the two materials is the path spec. One is "includes" and the other is "ignore" (implemented using the path spec). Does this result in it being two logically different materials?

You're right about fan-in. I guess my (currently unrealistic) expectation was that it would still be the same logical material, and trigger fan-in when a commit spans across multiple path specs.

There are multiple errors at the moment, one for each material name:

Cannot save SCM, found multiple SCMs called 'my-repo'. SCM names are case-insensitive and must be unique.
Cannot save SCM, found multiple SCMs called 'my-repo2'. SCM names are case-insensitive and must be unique.

from gocd-git-path-material-plugin.

Are you using the yaml or json config repo plugins? Or hand crafting via the UI? I'd probably need to see raw config (the yaml, or the exported pipeline definitions if hand crafting) to figure out what GoCD is complaining about.

When you delete a pipeline via UI i think it doesn't delete old SCMs linked via materials so there might be some cruft behind leading to that error.

from gocd-git-path-material-plugin.

But yes, to answer your earlier question, those will be different materials with the git-path plugin. That's what you want normally.

If you want them to be considered the same material you can just use whitelists and blacklists with the built-in gocd plugin. But depending on how you structure your pipelines, if you do want fan-in, you will see why it's not a good idea - you'll have fan-in pipelines that refuse to trigger, because the upstream pipelines weren't triggered with the same revision off the repo.

The main goal of this plugin is really to ensure the whitelist/blacklist (include/exclude) contributes to the uniqueness of the material, in addition to the pipeline triggering. You can think of this plugin like hash(url, username, pathspec) whereas the built-in plugin is hash(url, username) regardless of whitelist/blacklist configuration.

from gocd-git-path-material-plugin.

We are defining pipelines using the YAML plugin. Here's excerpts from both the pipelines:

pipelines:
  helm-package-publish:
    group: my-group
    label_template: ${COUNT}
    lock_behavior: none
    display_order: -1
    materials:
      my-repo:
        plugin_configuration:
          id: git-path
        options:
          url: [email protected]:myorganization/myrepo.git
          path: helm/**/*, helm.gocd.yaml
---
  build-test-publish:
    group: personalization
    label_template: ${COUNT}
    lock_behavior: none
    display_order: -1
    materials:
      my-repo2:
        plugin_configuration:
          id: git-path
        options:
          url: [email protected]:myorganization/myrepo.git
          path: :(exclude)helm/**/*, :(exclude)helm.gocd.yaml 

When you delete a pipeline via UI i think it doesn't delete old SCMs linked via materials so there might be some cruft behind leading to that error.

I think this is part of the problem. Yesterday, it was showing errors for more than one config repositories, and today it's showing errors for just one. We haven't changed any pipeline definition since yesterday. Probably a bug in GoCD.

from gocd-git-path-material-plugin.

I think this is part of the problem. Yesterday, it was showing errors for more than one config repositories, and today it's showing errors for just one. We haven't changed any pipeline definition since yesterday. Probably a bug in GoCD.

Yeah, I have also seen some weird stuff with validation of config repos and pipelines defined by the yaml plugin. Maybe check you dont have two config repos configured that are searching for the same pipeline yaml; or a duplicated pipeline yaml?

from gocd-git-path-material-plugin.

The error went away after two days 😒 I think we can close this. Thank you @chadlwilson!

from gocd-git-path-material-plugin.