Support for webhooks triggers of pipeline about gocd-git-path-material-plugin HOT 8 CLOSED

No haven't tried this scenario

from gocd-git-path-material-plugin.

Hi @maxbalan - thanks for trying out v2!

Are you able to share the (partially redacted) configuration you're using (SSH vs HTTPS, pipeline config for the materials) and any errors you see from either the webhook calls or in GoCD server logs?

I must admit that I have never tried the webhooks with GoCD and I'm not sure if they are designed for integration with custom SCM materials served by plugins. The custom material API this plugin uses hasn't been changed in a very long time; since GoCD 15.1.0 whereas the webhook support is more recent (2017?).

I understand GoCD uses some matching logic to determine which material update to fire based on an incoming webhook event, but not sure

• how that matching would work with plugin-based materials
• moreso how it would work specifically with this plugin since the trigger would presumably inform the plugin of a revision that has been pushed; which the plugin would then need to re-check to determine whether any of the pushed commits have changed the monitored paths it is interested in; and potentially change the revision or report to the GoCD server that the material has not actually changed if the pushed commits were for irrelevant paths.

So we'd need to work out

• is triggering plugin-based materials from webhooks supported by GoCD?
• if so, is the plugin doing what is necessary to support this?

from gocd-git-path-material-plugin.

Hi @chadlwilson thank you for such a quick reply, I did presume that the issue lies down deeper in the code, unfortunately because of lack of time I did not had a chance to look into source code.

In our case this is a big issue as we have lots and lots of projects and as you may know polling for change has a limitations, provider like github and bitbucket limit this types of request to a low number, and in situation like this when the company has lots of repos this limitation is reached in minutes, and the only solution we have is to use webhooks.

Though we changed a lot of the repos to webhook there are still too many monorepos that still triggers the backoff policy for polling which means we are not able to build and release it in automated way for the next hour or so.

I think a feature like this will be a huge win for the plugin, I will do my best to assist you with anything I can.

I disabled polling for the pipeline and pushed a change, here are the logs you've requested,

gocd logs

INFO   | jvm 1    | 2020/05/21 11:33:25 | 2020-05-21 11:33:25,191 INFO  [qtp415774130-41] PushWebhookControllerV1:110 - [WebHook] Noticed a git push to [REPO_NAME] on branch master.

bitbucket webhook response

    HTTP status: 202
    Elapsed time: 187ms
    Request time: 2 minutes ago (Thursday, May 21st 2020, 12:33:25 pm)
    Response Body
        {
          "message" : "No matching materials!"
        }

from gocd-git-path-material-plugin.

Hi @maxbalan - yeah, I certainly understand the desire for this. In my current use cases we have dozens of materials polling an on-prem BitBucket Server and it both increases lags for triggering after a push (do to so many materials competing to poll) and increases load on the BitBucket Server.

Unfortunately, as far as I can see, the GoCD Server support is not intended to support plugins (which are PluggableSCMMaterials) for webhook invocations.

The searches for materials are hardcoded to only match the built-in GitMaterial.

So at the very least a GoCD Server enhancement and change to the SCM material API would be needed to support this, which I've raised at gocd/gocd#8170 and will have a think about.

from gocd-git-path-material-plugin.

@chadlwilson Thank you for looking into this request, please let me know if I can help with anything

from gocd-git-path-material-plugin.

Hi @maxbalan - in theory this may be supported from GoCD 20.9.0. I haven't tried it myself with this plugin yet, but wondering if you happen to have attempted it? 👍

from gocd-git-path-material-plugin.

hi @chadlwilson sorry for such a late reply, must have missed the notification. I was actually following the progress of this feature and currently upgraded to gocd 21.1.0 which has the change for a while now, I did migrate the repos, and it looks good, everything seems to be working like a charm.
Thank you and everyone who worked on this feature 🥇

from gocd-git-path-material-plugin.

Thanks for the reply - that's great to hear.

Did you happen to try cases where you make a single commit and you'd expect it to trigger say two different pipelines, filtering on different material path expressions within the same repo; but affected by that single commit?

from gocd-git-path-material-plugin.