Argo version: 2.4.10
Argo helm chart: 4.10.7

Problem Statement

I'm working on using ArgoCD Notifications to send deployment notifications to slack. I successfully set it up to notify a slack channel on deployment and I have set up the trigger to only on new revisions according to this doc. This works great 👍

All of our Application objects live in repo A and they point to helm charts for those applications in repo B on the master branch. When we do a deploy, we update the version in a helm values.yaml file in repo B for the appropriate chart. Because this repo is shared between all our applications, argo triggers a deploy notification (and a sync) for all applications. We want to limit these notifications only to the application being deployed. Unfortunately, the oncePer: app.status.sync.revision refers to the commit hash of repo B. I'd like to change this to refer to something outside the app context.

Possible solution?

The docs refer to a few additional bits of data we should have access to in both triggers and templates. Accoding to the docs, there should be some helm context available. If I were able to use the version I set in my helm values, I could avoid having to restructure my organization's entire codebase.

I can test this in a slack template and, as best I can understand it, try to fetch the image.tag value from the helm values.

          {
            "title": "Helm Version",
            "value": "{{repo.GetAppDetails().GetParameterValueByName(\"image.tag\")}}",
            "short": true
          },

However, I get this error when running the template: failed to create API: template: app-deployed:23: function "repo" not defined.

Upon further investigation, I'm unable to find anything that would expose this in the code or any of the other documented functions. Admittedly, my golang isn't strong so I may be wrong here. It looks like the context that is passed to the expr package that compiles these statements only includes the Application object context as app. See https://github.com/argoproj/notifications-engine/blob/master/pkg/cmd/trigger.go#L61 and https://github.com/argoproj/notifications-engine/blob/master/pkg/cmd/template.go#L58

What now?

I'm blocked. When I inspect the Application metadata in Kubernetes, the only thing I'm able to find that may differentiate different deployments is app.status.summary.images[]. It would be sufficient for now if I could use this to populate oncePer in the trigger. However, our devs will eventually want to notify for any deployment unrelated to the image (like a values.yaml update).

Is there something I'm missing? Is there anything as a consumer I can do to make this work?

Currently, when sending github notifications, it is always tied to the current sync revision. However, to be accurate, it should be tied to the revision for the specific revision, app.status.operationState.syncResult.revision

When running in golang 1.16 all tests succeed:

$ docker run --rm -v ${PWD}:/src golang:1.16.13 '/bin/bash' '-c' 'cd /src && make test'
go test ./... -coverprofile=coverage.out -race
go: downloading k8s.io/api v0.20.4
... (lots of modules)
go: downloading golang.org/x/text v0.3.4
?   	github.com/argoproj/notifications-engine/docs/services	[no test files]
?   	github.com/argoproj/notifications-engine/examples/certmanager/cli	[no test files]
?   	github.com/argoproj/notifications-engine/examples/certmanager/controller	[no test files]
ok  	github.com/argoproj/notifications-engine/pkg/api	1.246s	coverage: 66.4% of statements
ok  	github.com/argoproj/notifications-engine/pkg/cmd	0.111s	coverage: 58.6% of statements
ok  	github.com/argoproj/notifications-engine/pkg/controller	0.290s	coverage: 76.2% of statements
?   	github.com/argoproj/notifications-engine/pkg/docs	[no test files]
?   	github.com/argoproj/notifications-engine/pkg/mocks	[no test files]
ok  	github.com/argoproj/notifications-engine/pkg/services	0.108s	coverage: 55.5% of statements
?   	github.com/argoproj/notifications-engine/pkg/services/mocks	[no test files]
ok  	github.com/argoproj/notifications-engine/pkg/subscriptions	0.033s	coverage: 62.4% of statements
ok  	github.com/argoproj/notifications-engine/pkg/templates	0.029s	coverage: 83.3% of statements
ok  	github.com/argoproj/notifications-engine/pkg/triggers	0.024s	coverage: 82.4% of statements
?   	github.com/argoproj/notifications-engine/pkg/util/http	[no test files]
?   	github.com/argoproj/notifications-engine/pkg/util/misc	[no test files]
ok  	github.com/argoproj/notifications-engine/pkg/util/slack	0.034s	coverage: 85.2% of statements
?   	github.com/argoproj/notifications-engine/pkg/util/slack/mocks	[no test files]
?   	github.com/argoproj/notifications-engine/pkg/util/text	[no test files]

While in golang 1.17 the tests that use reflection fails:

$ docker run --rm -v ${PWD}:/src golang:1.17.6 '/bin/bash' '-c' 'cd /src && make test' 
go test ./... -coverprofile=coverage.out -race
go: downloading github.com/spf13/cobra v1.1.3
... (lots of modules)
go: downloading golang.org/x/text v0.3.4
?   	github.com/argoproj/notifications-engine/docs/services	[no test files]
?   	github.com/argoproj/notifications-engine/examples/certmanager/cli	[no test files]
?   	github.com/argoproj/notifications-engine/examples/certmanager/controller	[no test files]
ok  	github.com/argoproj/notifications-engine/pkg/api	1.194s	coverage: 66.4% of statements
ok  	github.com/argoproj/notifications-engine/pkg/cmd	0.089s	coverage: 58.6% of statements
ok  	github.com/argoproj/notifications-engine/pkg/controller	0.288s	coverage: 76.2% of statements
?   	github.com/argoproj/notifications-engine/pkg/docs	[no test files]
?   	github.com/argoproj/notifications-engine/pkg/mocks	[no test files]
time="2022-01-28T03:40:38Z" level=error msg="alertmanager sent target: Post \"http://127.0.0.1:29093/api/v2/alerts\": dial tcp 127.0.0.1:29093: connect: connection refused" service=alertmanager
time="2022-01-28T03:40:38Z" level=error msg="alertmanager sent target: Post \"http://127.0.0.1:39093/api/v2/alerts\": dial tcp 127.0.0.1:39093: connect: connection refused" service=alertmanager
time="2022-01-28T03:40:38Z" level=error msg="alertmanager sent target: Post \"http://127.0.0.1:19093/api/v2/alerts\": dial tcp 127.0.0.1:19093: connect: connection refused" service=alertmanager
--- FAIL: TestBuildMessageOptionsUsername (0.00s)
    slack_test.go:79: 
        	Error Trace:	slack_test.go:79
        	Error:      	"github.com/argoproj/notifications-engine/pkg/services.buildMessageOptions.func1" does not contain "MsgOptionUsername"
        	Test:       	TestBuildMessageOptionsUsername
--- FAIL: TestBuildMessageOptionsIcon (0.00s)
    slack_test.go:92: 
        	Error Trace:	slack_test.go:92
        	Error:      	"github.com/argoproj/notifications-engine/pkg/services.buildMessageOptions.func2" does not contain "MsgOptionIconEmoji"
        	Test:       	TestBuildMessageOptionsIcon
FAIL
coverage: 55.5% of statements
FAIL	github.com/argoproj/notifications-engine/pkg/services	0.052s
?   	github.com/argoproj/notifications-engine/pkg/services/mocks	[no test files]
ok  	github.com/argoproj/notifications-engine/pkg/subscriptions	0.051s	coverage: 62.4% of statements
ok  	github.com/argoproj/notifications-engine/pkg/templates	0.036s	coverage: 83.3% of statements
ok  	github.com/argoproj/notifications-engine/pkg/triggers	0.037s	coverage: 82.4% of statements
?   	github.com/argoproj/notifications-engine/pkg/util/http	[no test files]
?   	github.com/argoproj/notifications-engine/pkg/util/misc	[no test files]
--- FAIL: TestThreadedClient (0.00s)
    --- FAIL: TestThreadedClient/Post,_basic (0.00s)
        client.go:168: Unexpected call to *mocks.MockSlackClient.SendMessageContext([context.TODO channel 0x6c1ba0 0x68b3c0]) at /src/pkg/util/slack/client.go:168 because: 
            expected call at /src/pkg/util/slack/client_test.go:158 doesn't match the argument at index 2.
            Got: [0x6c1ba0 0x68b3c0] ([]interface {})
            Want: MsgOptionPost
        controller.go:269: missing call(s) to *mocks.MockSlackClient.SendMessageContext(is anything, is anything, MsgOptionPost) /src/pkg/util/slack/client_test.go:158
        controller.go:269: aborting test due to missing call(s)
    --- FAIL: TestThreadedClient/Post,_no_parent,_with_grouping (0.00s)
        client.go:168: Unexpected call to *mocks.MockSlackClient.SendMessageContext([context.TODO channel 0x6c1ba0 0x68b3c0]) at /src/pkg/util/slack/client.go:168 because: 
            expected call at /src/pkg/util/slack/client_test.go:158 doesn't match the argument at index 2.
            Got: [0x6c1ba0 0x68b3c0] ([]interface {})
            Want: MsgOptionPost
        controller.go:269: missing call(s) to *mocks.MockSlackClient.SendMessageContext(is anything, is anything, MsgOptionPost) /src/pkg/util/slack/client_test.go:158
        controller.go:269: aborting test due to missing call(s)
    --- FAIL: TestThreadedClient/Post,_with_parent,_with_grouping (0.00s)
        client.go:168: Unexpected call to *mocks.MockSlackClient.SendMessageContext([context.TODO channel 0x6c1ba0 0x68b3c0]) at /src/pkg/util/slack/client.go:168 because: 
            expected call at /src/pkg/util/slack/client_test.go:158 doesn't match the argument at index 2.
            Got: [0x6c1ba0 0x68b3c0] ([]interface {})
            Want: MsgOptionPost
        controller.go:269: missing call(s) to *mocks.MockSlackClient.SendMessageContext(is anything, is anything, MsgOptionPost) /src/pkg/util/slack/client_test.go:158
        controller.go:269: aborting test due to missing call(s)
    --- FAIL: TestThreadedClient/PostAndUpdate,_no_parent._First_post_should_not_be_updated (0.00s)
        client.go:168: Unexpected call to *mocks.MockSlackClient.SendMessageContext([context.TODO channel 0x6c1ba0 0x68b3c0]) at /src/pkg/util/slack/client.go:168 because: 
            expected call at /src/pkg/util/slack/client_test.go:158 doesn't match the argument at index 2.
            Got: [0x6c1ba0 0x68b3c0] ([]interface {})
            Want: MsgOptionPost
        controller.go:269: missing call(s) to *mocks.MockSlackClient.SendMessageContext(is anything, is anything, MsgOptionPost) /src/pkg/util/slack/client_test.go:158
        controller.go:269: aborting test due to missing call(s)
    --- FAIL: TestThreadedClient/PostAndUpdate,_with_parent._First_post_should_be_updated (0.00s)
        client.go:168: Unexpected call to *mocks.MockSlackClient.SendMessageContext([context.TODO channel 0x6c1ba0 0x68b3c0]) at /src/pkg/util/slack/client.go:168 because: 
            expected call at /src/pkg/util/slack/client_test.go:158 doesn't match the argument at index 2.
            Got: [0x6c1ba0 0x68b3c0] ([]interface {})
            Want: MsgOptionPost
            expected call at /src/pkg/util/slack/client_test.go:163 doesn't match the argument at index 2.
            Got: [0x6c1ba0 0x68b3c0] ([]interface {})
            Want: MsgOptionUpdate
        controller.go:269: missing call(s) to *mocks.MockSlackClient.SendMessageContext(is anything, is anything, MsgOptionPost) /src/pkg/util/slack/client_test.go:158
        controller.go:269: missing call(s) to *mocks.MockSlackClient.SendMessageContext(is anything, is anything, MsgOptionUpdate) /src/pkg/util/slack/client_test.go:163
        controller.go:269: aborting test due to missing call(s)
    --- FAIL: TestThreadedClient/Update,_no_parent._Only_call_should_be_post (0.00s)
        client.go:168: Unexpected call to *mocks.MockSlackClient.SendMessageContext([context.TODO channel 0x6c1ba0 0x68b3c0]) at /src/pkg/util/slack/client.go:168 because: 
            expected call at /src/pkg/util/slack/client_test.go:158 doesn't match the argument at index 2.
            Got: [0x6c1ba0 0x68b3c0] ([]interface {})
            Want: MsgOptionPost
        controller.go:269: missing call(s) to *mocks.MockSlackClient.SendMessageContext(is anything, is anything, MsgOptionPost) /src/pkg/util/slack/client_test.go:158
        controller.go:269: aborting test due to missing call(s)
    --- FAIL: TestThreadedClient/Update,_with_parent._Only_call_should_be_update (0.00s)
        client.go:168: Unexpected call to *mocks.MockSlackClient.SendMessageContext([context.TODO channel 0x6bf0e0 0x6bf040 0x68b3c0]) at /src/pkg/util/slack/client.go:168 because: 
            expected call at /src/pkg/util/slack/client_test.go:158 doesn't match the argument at index 2.
            Got: [0x6bf0e0 0x6bf040 0x68b3c0] ([]interface {})
            Want: MsgOptionUpdate
        controller.go:269: missing call(s) to *mocks.MockSlackClient.SendMessageContext(is anything, is anything, MsgOptionUpdate) /src/pkg/util/slack/client_test.go:158
        controller.go:269: aborting test due to missing call(s)
FAIL
coverage: 63.9% of statements
FAIL	github.com/argoproj/notifications-engine/pkg/util/slack	0.015s
?   	github.com/argoproj/notifications-engine/pkg/util/slack/mocks	[no test files]
?   	github.com/argoproj/notifications-engine/pkg/util/text	[no test files]
FAIL
make: *** [Makefile:3: test] Error 1

This is more of a heads-up since go.mod specifies 1.16 but it hit me as I'm using 1.17 for all the other projects.

Currently, the GitHub service only creates a commit status (code). It would be good if it also supports creating a comment to all associated pull requests.

Code snippet:

// create a comment on all referenced pull requests
prs, _, err := g.client.PullRequests.ListPullRequestsWithCommit(context.Background(),
	u[0],
	u[1],
	notification.GitHub.revision,
	nil,
)
commentBody := fmt.Sprintf("Sync for commit %q is %s", notification.GitHub.revision, notification.GitHub.Status.State)
for _, pr := range prs {
	prComment := &github.IssueComment{
		Body: &commentBody,
	}
	_, _, err := g.client.Issues.CreateComment(context.Background(),
		u[0],
		u[1],
		pr.GetNumber(),
		prComment,
	)
	if err != nil {
		return err
	}
}

Defined a context in the configmap and the notification was trigger. However the .context.fieldname always shows this is with Argo rollouts. Here is the configmap for the notification
...
data:
context: |
region: "east"
environmentName: "staging"

service.webhook.form: |
url: http://172.20.2.210:4566/00000000000/canary-status-queue
headers:
- name: Content-Type
value: application/x-www-form-urlencoded

template.rollout-completed: |
webhook:
form:
method: POST
body: Action=SendMessage&MessageBody={"environment":"{{ .context.environmentName }}","rollout":"{{.rollout.metadata.name}}","status":"completed"}

{{.context.xxx}} are in several sample templates, does this field only work for ArgoCD?

Background: I am trying to retrieve a sub-string of the application spec source path to pass to our CI workflow using sprig split function inside the notification template. This was working perfectly fine when we had a source path like this -> charts/app1 and his worked with this template

  template.chart-app-sync-succeeded-organization: |
    webhook:
      github-deploy-workflow:
        method: POST
        path: repos/foo/release-chart-staging.yaml/dispatches
        body: |
          {
            "ref": "main",
            "inputs": { "chart": "{{(split "/" .app.spec.source.path)._1}}", "type": "organization" }
          }

Problem: We have now moved our charts to sub-folders so I now need to access the second element of the split for example -> chats/global/app1

  template.chart-app-sync-succeeded-organization: |
    webhook:
      github-deploy-workflow:
        method: POST
        path: repos/foo/release-chart-staging.yaml/dispatches
        body: |
          {
            "ref": "main",
            "inputs": { "chart": "{{(split "/" .app.spec.source.path)._2}}", "type": "organization" }
          }

By changing the index I no longer am able to send Webhooks to GitHub and am receiving this error.

time="2022-04-13T02:59:42Z" level=error msg="failed to execute when condition: cannot fetch phase from (1:27)\n | app.status.operationState.phase in ['Succeeded']\n | ..........................^"

I checked what was being output by split and can clearly see the dict being generated as shown here:

time="2022-04-13T02:57:40Z" level=error msg="Failed to notify recipient {github-deploy-workflow } defined in resource argocd/us-central1-static-fhir-resources-api: request to {{\n "ref": "main",\n "inputs": { "chart": "map[_0:charts _1:organization _2:static-fhir-resources-api _3:dev]", "type": "organization" }\n}\n

I am at a loss here... any help would be great!

Summary

For webhook notification services, provide the ability to forward the entire Rollout as JSON.

Use Cases

We would like to decide which fields to use within our service that handles notification webhooks, rather than within the notification template. This would allow us to reduce the coupling between the template and the webhook handling service that consumes the notifications (which currently have to agree on the schema). It would also cut down on the boilerplate that exists in our template definitions, which looks like:

{
  "deployment_name": "{{.rollout.spec.workloadRef.name}}",
  "current_step_index": "{{.rollout.status.currentStepIndex}}",
  "replicas": "{{.rollout.status.replicas}}"
  ...
}

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

By @daurnimator: argoproj-labs/argocd-notifications#91

Our ops team primarily uses IRC to communicate.
We'd love to get ArgoCD notifications posted to a channel.

In argocd notifications, I use the following trigger to filter notifications with the oncePer option.

    trigger.on-deployed: |
      - description: Application is synced and healthy. Triggered once per commit.
        when: app.status.health.status == 'Healthy' and app.status.operationState != nil and app.status.operationState.phase in ['Succeeded']
        oncePer: app.status.operationState.syncResult.revision
        send:
        - app-deployed

This returns the following error in the logs

{"level":"error","msg":"failed to execute oncePer condition: cannot fetch syncResult from \u003cnil\u003e (1:27)\n | app.status.operationState.syncResult.revision\n | ..........................^","time":"2022-01-28T15:41:46Z"}

Solution

I think the oncePer condition should not be evaluated when the when condition evaluate to false.

The problem.
My teammates are installing its apps by argo from helm-chart-repository and we're not able to use notification-plugin to push statusCheck to github repository which we specify. GitHub Service only takes it directly from application spec and there is no other option to override this value.

Could you please make it possible?

Thanks!

By @GodSpeedXI: argoproj-labs/argocd-notifications#275

It would be great to support LINE notification. Some teams used to it. thanks

The mail integration have not supported text/html, only text/plain now.

from: argoproj-labs/argocd-notifications#291

By @tobru: argoproj-labs/argocd-notifications#233

It would be great to see notifications to Rocket.Chat directly integrated into Argo CD Notifications.

Running the argocd-notifications binary to test template is impossible when the application name contains a . in its name.

kubectl exec -it argocd-notifications-controller-744b5d8868-84xh4 -- /app/argocd-notifications template notify app-sync-succeeded my.application.name
failed to load resource: open my.application.name: no such file or directory

Problem

When the RESOURCE_NAME contains a ., it is assumed it's a file.

Problem Statement:

It appears the pagerduty.go service implementation recently used a rate-limited synchronous API intended for human initiated incidents.

Root Cause Analysis

The Notification Engine utilizes CreateIncidentWithContext from the PagerDuty Go SDK v1.5.0. Upon inspection of the implementation of CreateIncidentWithContext, it can be determined that the API Path /incident corresponds to PagerDuty's Incident Creation API. This has concerning implications when utilizing the API in correspondence with Machine Events due to the following exerpts from the provided API documentation:

This API is not to be used for connecting your monitoring tools to send events to PagerDuty; for that, use the Events v2
...
Unlike the Events APIs, the Incident Creation API is heavily rate limited on a per-account basis. It’s meant for the creation of events at "human speed" - in response to user action, rather than automated tooling.
...
This API is also synchronous, as it returns a fully-formed Incident object (rather than the incident key returned by the asynchronous Events APIs). The Events APIs accept arbitrary JSON objects, while the Incident creation API currently only supports string Incident bodies.
...
The Incident Creation API is analogous to clicking on the "Create Incident" button in PagerDuty’s web or mobile interfaces. It captures incident information from users and uses that to create a new incident, regardless of monitoring tool data. With this API, one can connect to a PagerDuty account and create or edit incidents on that account. This API is not to be used for connecting your monitoring tools to send events to PagerDuty; for that, use the Events v2

Git Blame:

The changeset introducing this code was authored by @RaviHari and merged by @pasha-codefresh . Additional discussion is necessary to identify whether there is consensus from the original author and approver on the problem statement and recommended solution.

Recommendation

Add

• Replace removed PagerDuty Incident Creation API support with the Asynchronous API recommendations from PagerDuty v2 Events API.

The Events API v2 is a highly reliable, highly available asynchronous API that ingests machine events from monitoring tools and other systems like code repositories, observability platforms, automated workflow tools, and configuration management systems.

Remove

• Remove support for PagerDuty Incident Creation API for subscriptions to machine generated events such as Rollout Aborted.

Alternative & Additional Solutioning

• Copy or modify Incident API into a ArgoProj repository that offers a User Interface (UI). Write UI logic to support a human initiated event that calls CreateIncidentWithContext as intended by the API.
• Expand adoption of PagerDuty v2 Events API to support the Asynchronous Change Events API, in addition to the Alert Events API

There is no Bitbucket integration for status update. The only way is to configure the webhook to send a REST request, but it only supports one single REST call.

If we want to create both a commit status and a comment to the associated pull requests, it's better to have a Bitbucket service.

When trying to send a notification to AWS SQS with webhook, the message will be rejected for missing the signature. Suggest add support for SQS. With AWS SKD, the signature will be taken care behind the scene

There is no GitLab integration for status update. The only way is to configure the webhook to send a REST request, but it only supports one single REST call.

If we want to create both a commit status and a comment to the associated merged requests, it's better to have a GitLab service.

Currently, the notification engine leverages the Logrus library to log errors. It is non-trivial to extract the errors for further processing.

One possible solution is to add a configurable callback to the notification controller. Here is a draft proposal: 4806f9a.

I’d like to get notifications on Discord with argocd-notifications. I tried pointing the engine’s slack service at the Slack-compatible webhook endpoint, but according to the trace-level logs, this returned a 404 because the actual request URL had chat.postMessage appended to it. In any case, Discord doesn’t support exactly the same things as Slack, so it’ll only work as long as the notifier doesn’t use these:

We do not support Slack's channel, icon_emoji, mrkdwn, or mrkdwn_in properties.

The existing webhook interface is another option, but it would be nice to have the engine provide a higher-level interface like it does for many other services.

EDIT:
I may have misunderstood the Slack– & GitHub-compatible endpoints. I think they’re for incoming webhooks from those services, and so irrelevant to this discussion.

It would be great to support PagerDuty notification. Some teams used to it!

Currently, the notification subscription annotation key is hard-coded with the notifications.argoproj.io prefix. It would be more flexible if we can use a different prefix in the annotation.

Same for the notified annotation.

Using notification controller bundled in argocd:v2.4.8 with config:

  service.alertmanager: |
    targets:
      - webhook.site:443
    scheme: https
    apiPath: /0be1114e-c211-4fb3-bb47-908e51aa9675/api/events

and app annotation

notifications.argoproj.io/subscribe.on-sync-succeeded.alertmanager: ""

I can't get any notifications on my endpoint. I can see this error in logs:

time="2023-01-06T20:59:45Z" level=error msg="Failed to notify recipient {alertmanager } defined in resource argocd/guestbook: notification alertmanager no config" resource=argocd/guestbook

At the moment its not possible to retrieve the GitHub appId or installationID from secrets at the moment as the decoding fails.

error unmarshaling JSON: json: cannot unmarshal string into Go struct field GitHubOptions.appID of type int64"

Summary

Bug: wrong Github Enterprise API URL is being used

Diagnostics

Setting up argo notification GitHub service:

1. Created new Github app and changed Commit Statuses pre-missions to read-write, saved the pem file and installed the GitHub app

2. set the config map:

service.github: |
       appID: <GITHUB_APP_ID>
       installationID: <GITHUB_INSTALLATION_ID>
       privateKey: $github_privateKey
       enterpriseBaseURL: https://git.<ENTERPRISE>.com

3. added a message:

template.<trigger>: |
  message: |
    Application {{.app.metadata.name}} deploy-started.
  github:
    status:
      state: started
      label: "continuous-delivery/{{.app.metadata.name}}"
      targetURL: "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}?operation=true

4. created the secret same as the Docs

5. adde the annotation:
   notifications.argoproj.io/subscribe.<trigger>.github: {{ <REPO_NAME> | quote }}

POST command used by argo notification in order to connect with the GitHub app that we created:
POST URL:https://git.<ENTERPRISe>.com/app/installations/<INSTALLATION_ID>/access_tokens

The response for that POST was: “non 2xx response and a redirect”

received non 2xx response status {\"<html><body>You are being <a href=\\\"https://git.<ENTERPRISE>.com/login?return_to=https%3A%2F%2Fgit.<ENTERPRISE>.com%2Fapp%2Finstallations%2F<INSTALLATION_ID>%2Faccess_tokens\\\">redirected</a>.</body></html>\"}

Our lead was checking the Bearer token so we tested it locally (followed the Authenticating with GitHub Apps - Doc)

Steps to reproduce the error:

Generate the fingerprint of your private key (PEM) locally by using the following command:
openssl rsa -in PATH_TO_PEM_FILE -pubout -outform DER | openssl sha256 -binary | openssl base64

No diff was found when compared the results of the locally generated fingerprint to the fingerprint of the private key in GitHub.

Created the Bearer token locally using the following ruby script:

YOUR_PATH_TO_PEM and YOUR_APP_ID are the values you must replace. Make sure to enclose the values in double quotes.

require 'openssl'
require 'jwt'  # https://rubygems.org/gems/jwt

// Private key contents
private_pem = File.read("YOUR_PATH_TO_PEM")
private_key = OpenSSL::PKey::RSA.new(private_pem)

// Generate the JWT
payload = {
  // issued at time, 60 seconds in the past to allow for clock drift
  iat: Time.now.to_i - 60,
  // JWT expiration time (10 minute maximum)
  exp: Time.now.to_i + (10 * 60),
 // GitHub App's identifier
  iss: "YOUR_APP_ID"
}

jwt = JWT.encode(payload, private_key, "RS256")
puts jwt

Trying to POST to GitHub, with the same URL that was used by argo notification

Error Reproduced

$ curl -i -X POST \
-H "Authorization: Bearer $JWT" \
-H "Accept: application/vnd.github.v3+json" \
https://git.<ENTERPRISE>.com/app/installations/<INSTALLATIN_ID>/access_tokens

We got the same error message that we saw in the log of argo notification.

When changing the url to the correct github enterprise api url

from: https://git.<ENTERPRISE>.com/app/installations/<INSTALLATION_ID>/access_tokens
to: https://git.<ENTERPRISE>.com/api/v3/app/installations/<INSTALLATION_ID>/access_tokens

curl -i -X POST \
-H "Authorization: Bearer $JWT" \
-H "Accept: application/vnd.github.v3+json" \
https://git.<ENTERPRISE>.com/api/v3/app/installations/<INSTALLATIN_ID>/access_tokens

We were able to connect with our GitHub app.

Full Error Log:
time="2022-05-01T09:05:06Z" level=error msg="Failed to notify recipient {github <REPO_NAME>} defined in resource argo/<APP_NAME>: Post \"https://git/.<Enterprise>.com/api/v3/repos/<REPO_NAME>/<CHART_NAME>/statuses/<REVISION>\": could not refresh installation id <INSTALATION_ID>'s token: request &{Method:POST URL:https://git/.<Enterprise>.com//app/installations/<INSTALATION_ID>/access_tokens Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept:[application/vnd.github.machine-man-preview+json application/vnd.github.machine-man-preview+json] Authorization:[Bearer <TOKEN>] Content-Type:[application/json]] Body:0xc00039e2b8 GetBody:0x7592e0 ContentLength:5 TransferEncoding:[] Close:false Host:git.<Enterprise>.com Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil> Cancel:<nil> Response:<nil> ctx:0xc000058018} received non 2xx response status {\"<html><body>You are being <a href=\\\"https://git/.<Enterprise>.com/login?return_to=https%3A%2F%2Fgit.<Enterprise>.com%2Fapp%2Finstallations%2F<INSTALATION_ID>%2Faccess_tokens\\\">redirected</a>.</body></html>\"} with body &{Method:POST URL:https://git/.<Enterprise>.com//app/installations/<INSTALATION_ID>/access_tokens Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept:[application/vnd.github.machine-man-preview+json application/vnd.github.machine-man-preview+json] Authorization:[Bearer <TOKEN>] Content-Type:[application/json]] Body:0xc00039e2b8 GetBody:0x7592e0 ContentLength:5 TransferEncoding:[] Close:false Host:git.<Enterprise>.com Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil> Cancel:<nil> Response:<nil> ctx:0xc000058018} and TLS &{Version:772 HandshakeComplete:true DidResume:false CipherSuite:4865 NegotiatedProtocol:h2 NegotiatedProtocolIsMutual:true ServerName:git.<Enterprise>.com PeerCertificates:[0xc000854b00 0xc000855080] VerifiedChains:[[0xc000854b00 0xc000855080 0xc000720c00]] SignedCertificateTimestamps:[] OCSPResponse:[48 130 1 20 10 1 0 160 130 1 13 48 130 1 9 6 9 43 6 1 5 5 7 48 1 1 4 129 251 48 129 248 48 129 158 162 22 4 20 165 206 55 234 235 176 117 14 148 103 136 180 69 250 217 36 16 135 150 31 24 15 50 48 50 50 48 52 51 48 48 48 53 55 48 49 90 48 115 48 113 48 73 48 9 6 5 43 14 3 2 26 5 0 4 20 18 215 139 64 44 53 98 6 250 130 127 142 216 146 36 17 180 172 245 4 4 20 165 206 55 234 235 176 117 14 148 103 136 180 69 250 217 36 16 135 150 31 2 16 8 72 124 32 138 115 52 97 43 184 227 28 182 128 174 208 128 0 24 15 50 48 50 50 48 52 51 48 48 48 52 50 48 49 90 160 17 24 15 50 48 50 50 48 53 48 54 50 51 53 55 48 49 90 48 10 6 8 42 134 72 206 61 4 3 2 3 73 0 48 70 2 33 0 209 201 6 60 251 230 149 144 10 19 124 176 74 240 167 150 74 163 25 212 123 156 98 83 250 36 133 150 19 216 164 56 2 33 0 220 226 67 245 146 74 149 44 244 65 157 220 6 205 206 100 24 145 89 56 73 38 78 117 142 63 133 94 6 74 234 211] TLSUnique:[] ekm:0x6a31a0}" resource=argo/<APP_NAME>

Also manipulating the enterpriseURL to fix the api call:
We tried to change the enterpriseBaseURL: https://git..com
to enterpriseBaseURL: https://git..com/api/v3 but we got the following error:

404 Not Found []

github.com/dgrijalva/jwt-go
CVE-2020-26160 (high)

the vulnerability affects both CD & Rollout projects.

argoproj/argo-rollouts#1784

Please support markdown parse mode with Telegram

Hi! I would like to send notification to our private Rocket chat server and I got error:

x509: certificate signed by unknown authority

Could tell me how to modify https://github.com/argoproj/notifications-engine/blob/master/pkg/services/rocketchat.go for allow insecure connection or how to add cert into pod?

I am getting index out of the range for accessing the second part of reporURL in here

I think repoURL is hard-coded to be .app.spec.source.repoURL in here which is argocd only and doesnt support argo rollouts?

I setup the notification-engine with the argo-rollouts deployment; everything looks good and messages are delivered to the slack channel as expected. However, I noticed that there are several error message each time some event is triggered:

INFO[2021-09-26T23:42:29-04:00] Start processing                              resource=default/rollouts-demo
INFO[2021-09-26T23:42:29-04:00] Reconciliation completed                      generation=3 namespace=default resourceVersion=2148175 rollout=rollouts-demo time_ms=15.550582
INFO[2021-09-26T23:42:29-04:00] Started syncing rollout                       generation=3 namespace=default resourceVersion=2148177 rollout=rollouts-demo
ERRO[2021-09-26T23:42:29-04:00] Failed to process: Object 'Kind' is missing in '{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"argoproj.io/v1alpha1\",\"kind\":\"Rollout\",\"metadata\":{\"annotations\":{\"notifications.argoproj.io/subscribe.on-rollout-completed.slack\":\"sdpl-test\",\"notifications.argoproj.io/subscribe.on-rollout-step-completed.slack\":\"sdpl-test\",\"notifications.argoproj.io/subscribe.on-rollout-updated.slack\":\"sdpl-test\"},\"name\":\"rollouts-demo\",\"namespace\":\"default\"},\"spec\":{\"replicas\":5,\"revisionHistoryLimit\":2,\"selector\":{\"matchLabels\":{\"app\":\"rollouts-demo\"}},\"strategy\":{\"canary\":{\"steps\":[{\"setWeight\":20},{\"analysis\":{\"args\":[{\"name\":\"duration\",\"value\":\"1s\"},{\"name\":\"exit-code\",\"value\":\"0\"}],\"templates\":[{\"templateName\":\"sleep-job\"}]}},{\"pause\":{}},{\"setWeight\":40},{\"pause\":{}},{\"setWeight\":60},{\"pause\":{}},{\"setWeight\":80},{\"pause\":{}}]}},\"template\":{\"metadata\":{\"labels\":{\"app\":\"rollouts-demo\"}},\"spec\":{\"containers\":[{\"image\":\"argoproj/rollouts-demo:blue\",\"name\":\"rollouts-demo\",\"ports\":[{\"containerPort\":8080,\"name\":\"http\",\"protocol\":\"TCP\"}],\"resources\":{\"requests\":{\"cpu\":\"5m\",\"memory\":\"32Mi\"}}}]}}}}\n","notifications.argoproj.io/subscribe.on-rollout-completed.slack":"sdpl-test","notifications.argoproj.io/subscribe.on-rollout-step-completed.slack":"sdpl-test","notifications.argoproj.io/subscribe.on-rollout-updated.slack":"sdpl-test","rollout.argoproj.io/revision":"3"},"creationTimestamp":"2021-09-27T02:28:28Z","generation":3,"managedFields":[{"apiVersion":"argoproj.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{},"f:notifications.argoproj.io/subscribe.on-rollout-completed.slack":{},"f:notifications.argoproj.io/subscribe.on-rollout-step-completed.slack":{},"f:notifications.argoproj.io/subscribe.on-rollout-updated.slack":{}}},"f:spec":{".":{},"f:replicas":{},"f:revisionHistoryLimit":{},"f:selector":{".":{},"f:matchLabels":{".":{},"f:app":{}}},"f:strategy":{".":{},"f:canary":{".":{},"f:steps":{}}},"f:template":{".":{},"f:metadata":{".":{},"f:labels":{".":{},"f:app":{}}},"f:spec":{".":{},"f:containers":{}}}}},"manager":"kubectl-client-side-apply","operation":"Update","time":"2021-09-27T02:28:28Z"},{"apiVersion":"argoproj.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{}}}}},"manager":"kubectl-argo-rollouts-darwin-amd64","operation":"Update","time":"2021-09-27T03:37:42Z"},{"apiVersion":"argoproj.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:rollout.argoproj.io/revision":{}}},"f:status":{".":{},"f:HPAReplicas":{},"f:availableReplicas":{},"f:blueGreen":{},"f:canary":{},"f:conditions":{},"f:currentPodHash":{},"f:currentStepHash":{},"f:currentStepIndex":{},"f:message":{},"f:observedGeneration":{},"f:phase":{},"f:readyReplicas":{},"f:replicas":{},"f:selector":{},"f:stableRS":{},"f:updatedReplicas":{}}},"manager":"rollouts-controller","operation":"Update","time":"2021-09-27T03:42:29Z"}],"name":"rollouts-demo","namespace":"default","resourceVersion":"2148177","uid":"f7377772-b6a9-41d4-baaa-f25bd7d670a4"},"spec":{"replicas":5,"revisionHistoryLimit":2,"selector":{"matchLabels":{"app":"rollouts-demo"}},"strategy":{"canary":{"steps":[{"setWeight":20},{"analysis":{"args":[{"name":"duration","value":"1s"},{"name":"exit-code","value":"0"}],"templates":[{"templateName":"sleep-job"}]}},{"pause":{}},{"setWeight":40},{"pause":{}},{"setWeight":60},{"pause":{}},{"setWeight":80},{"pause":{}}]}},"template":{"metadata":{"creationTimestamp":null,"labels":{"app":"rollouts-demo"}},"spec":{"containers":[{"image":"argoproj/rollouts-demo:red","name":"rollouts-demo","ports":[{"containerPort":8080,"name":"http","protocol":"TCP"}],"resources":{"requests":{"cpu":"5m","memory":"32Mi"}}}]}}},"status":{"HPAReplicas":5,"availableReplicas":4,"blueGreen":{},"canary":{},"conditions":[{"lastTransitionTime":"2021-09-27T03:37:42Z","lastUpdateTime":"2021-09-27T03:37:42Z","message":"Rollout is paused","reason":"RolloutPaused","status":"False","type":"Paused"},{"lastTransitionTime":"2021-09-27T03:42:28Z","lastUpdateTime":"2021-09-27T03:42:28Z","message":"RolloutCompleted","reason":"RolloutCompleted","status":"False","type":"Completed"},{"lastTransitionTime":"2021-09-27T03:42:29Z","lastUpdateTime":"2021-09-27T03:42:29Z","message":"Rollout does not have minimum availability","reason":"AvailableReason","status":"False","type":"Available"},{"lastTransitionTime":"2021-09-27T03:37:42Z","lastUpdateTime":"2021-09-27T03:42:29Z","message":"ReplicaSet \"rollouts-demo-6b5cbf7465\" is progressing.","reason":"ReplicaSetUpdated","status":"True","type":"Progressing"}],"currentPodHash":"6b5cbf7465","currentStepHash":"89b96d8b9","currentStepIndex":0,"message":"more replicas need to be updated","observedGeneration":"3","phase":"Progressing","readyReplicas":4,"replicas":5,"selector":"app=rollouts-demo","stableRS":"75f756d67b","updatedReplicas":1}}'  resource=default/rollouts-demo

The above is the argo-rollouts controller's log, where I think the error message of the last line comes from the notification-engine. It could be

Although not harmful to the rollouts CR, the error messages are annoying. Do I miss anything in the configuration step? Thanks.

summary

To send form-data can become complex as many fields, encode, readability, and so on.

example

apiVersion: v1
kind: ConfigMap
metadata:
  name: <config-map-name>
data:
  template.form-data: |
    webhook:
      <webhook-name>:
        method: POST
        form:
          key1: value1
          key2: value2

I’m trying to avoid receiving false alerts/spam-like notifications when applications' health status change briefly e.g degraded/unknown <-> healthy often within a minute.

I tried to add the following time condition to alert if the app has been in the current state for 5 mins or more but it’s not working, or maybe I’m using the wrong field app.status.operationState.startedAt here?

    trigger.on-health-degraded: |
      - description: Application has degraded
        oncePer: app.status.operationState.syncResult.revision
        send:
        - app-health-degraded
        when: app.status.health.status == 'Degraded' and time.Now().Sub(time.Parse(app.status.operationState.startedAt)).Minutes() >= 5

I'm curious if anyone solved this issue using argocd-notifications?

Same as #82

error unmarshaling JSON: json: cannot unmarshal string into Go struct field EmailOptions.port of type int

By @evgmoskalenko: argoproj-labs/argocd-notifications#234

It would be great to see notifications to the Facebook workplace chat group will be integrated into Argo CD Notifications

Is there any example I can trigger the notification with a webhook or integrated into a message queue?
The idea is to use the engine as a general-purpose notification service.

Or is it achieved by adding custom Triggers? Like

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
data:
  trigger.on-api-call: |
    - when: true
      send: [api-call-recevied]

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
data:
  template.api-call-recevied: |
    message: |
      You have received an API call with message {{get-api-request-body_or_param}}.

So this engine can be used behind a rest microservice.

Support for adding New Relic as a notification service for ArgoCD notifications. New Relic has the concept of Deployment Markers that allows us to register a custom event as a deployment to be correlated with other observability metrics.

By adding this integration anyone using New Relic can easily rely on it to automatically create deployment markers upon application deployments. Within the GitOps model this will also yield markers for configuration changes beyond regular application code change which makes for a pretty holistically approach for production management and observability.

It can look something like this

apiVersion: v1
kind: ConfigMap
metadata:
  name: <config-map-name>
data:
  service.mattermost: |
    apiURL: <api-url>
    apiKey: $newrelic-apiKey

apiVersion: v1
kind: Secret
metadata:
  name: <secret-name>
stringData:
  newrelic-apiKey: apiKey

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    notifications.argoproj.io/subscribe.<trigger-name>.newrelic: <app-id>

By @AlexDHoffer: argoproj-labs/argocd-notifications#240

Some existing CD systems e.g. Jenkins can post build results to Jira issues as well as natively link commits in the build to those issues. See here for details: https://plugins.jenkins.io/jira/

This is a slick feature for a CD system and since Atlassian's Jira is in such widespread usage these days it would be cool to have something similar, where ArgoCD can key off a Jira issue tag in a commit message and post when a successful sync of that tag occurred back to the issue itself in Jira.

An URL Parse error was output in a notification-controller's log, when I used Alertmanager notification service.

level=error msg="Failed to notify recipient {alertmanager } defined in resource xxx: parse \"[email protected]:xxx/yyy.git\": first path segment in URL cannot contain colon" resource=...

I think this if-statement returns an error because my repo url is ssh format (e.g. [email protected]:org/repo.git).

Therefore, we have to add a checker for ssh format git url, or convert it to http format if AlertmanagerNotification.GeneratorURL must be http format.

Summary

Mastodon is free and open-source software for running self-hosted social networking services.

Motivation

Mastodon will become one of the leading social networks in the industry. So, there is considerable interest in moving from Twitter to Mastodon.

Proposal

There is a Go SDK of the Mastodon API¹, so we can add Mastodon as a new notification channel.

Similar work has been done in the GoReleaser project².

/cc @Dentrax

The current Opsgenie service is limited to sending message, and description. It would be great to be able to send additional optional fields especially priority, and tags, as well as dynamically setting the responders at the point the alert is generated.

I would be willing to contribute a PR for this functionality if there is interest in it.

Hello, I have been developing and debugging my own notification-controller.

Describe the bug

As the following document says, when I debug my own controller secret can't read anymore.

https://github.com/argoproj/notifications-engine/tree/master/examples/certmanager#debugging-tools

go run mydir/debug/main.go template get --config-map ./my-cm.yaml --secret ./my-secret.yaml

• my-cm.yaml

apiVersion: v1
data:
  service.slack: |
    token: $slack-token
  template.app-deployed: |
    email:
      subject: Test
    message: |
      Test Message
 manifests.
  trigger.on-deployed: |
    - description: test
      send:
      - app-deployed
      when: pod.status.phase == 'Running'
kind: ConfigMap
metadata:
  creationTimestamp: null
  name: my-cm

• my-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
type: Opaque
data:
  slack-token: aG9nZWhvZ2U=

Actual behivior

the following error occurs.

failed to get api: file './my-secret.yaml' does not have '/Secret/my-secret'

Expected behivior

list is outputed.

NAME          PREVIEW
app-deployed  Test Message

What makes the actual behivior

As the following code, even though file is Secret this try to read ConfigMap.

I found out the reason so I will make the PR related to this issue!

ref: argoproj/argo-rollouts#1783

Notification engine should handle multiple triggers in a given annotation:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  annotations:
    notifications.argoproj.io/subscribe.on-rollout-completed,on-rollout-updated,on-rollout-pausedon-rollout-aborted,on-rollout-step-completed,on-scaling-replica-set.slack: ar-notifications-test

Irrespective of configuration, the webhook service always uses the GET method.

I found the culprit here:

By @aslafy-z: argoproj-labs/argocd-notifications#243

It would be great to see notifications to Google Chat directly integrated into Argo CD Notifications

I'm not sure if this issue should be submitted in this repo or in argocd-notifications, hopefully here is ok.

Currently the github service supports only the Create Status api.

Notifications to other GitHub API endpoints can be achieved using the generic webhook service, but this method does not support GitHub App Authentication.

GitHub App authentication is preferable to me because at my organization it can be quite difficult to obtain a GitHub machine/service user account, but it is quite easy to receive a new GitHub App instance.

Please provide the ability in the GitHub service to build custom notifications to GitHub. I believe the majority, if not all, of the GitHub REST API could be supported with the current implementation of the webhook service, so a similar (or exact duplicate) implementation but capable of authenticating using a GitHub App would be greatly appreciated.

There are 2 possible implementation options that come to mind:

1. Add a new method in the GitHub service that is capable of doing custom webhook invocations
2. Add a GitHub App authentication method to the generic webhook service

We were setting up our Grafana service but did not specify a message field in our notification template. The Grafana annotations were not being created as it was causing the following error in Grafana:
2022-02-09T20:21:11.649902048Zt=2022-02-09T20:21:11+0000 lvl=eror msg="Failed to save annotation" logger=context userId=0 orgId=1 uname= error="text field should not be empty" remote_addr=<IP address>

Expected

ArgoCD logs an error when Grafana returns a 400 error from its create annotation API.

Actual

ArgoCD reported that the notification was sent successfully

"time="2022-02-09T20:21:11Z" level=info msg="Sending notification about condition 'on-deployed.[0].y7b5sbwa2Q329JYH755peeq-fBs' to '{grafana tenant-api}'" resource=score-argo/beehive-sandbox"
timestamp: "2022-02-09T20:21:11.614583720Z"

Request

Adding a templates section to the documentation for Grafana notifications, similar to the docs for Slack or email notifications, could be helpful. It could explicitly state that notification template requires specifying the message field for Grafana annotations.

It could be helpful for this method to check the status code of the Grafana response and return an error if the annotation was not created. It could also check whether message is an empty string and send an error message if it is.

It’d be great to add additional functionality to the GitHub integration so that it supports creating/updating deployments using the deployments API.

https://docs.github.com/en/rest/deployments/deployments

i create a template for a message to slack:

templates: 
  template.test: |
    slack:
      attachments: |
        [{
          "blocks": [
            {
              "type": "section",
              "text": {
                "type": "mrkdwn",
                "text": "<{{.app.spec.source.repoURL}}/commit/{{.app.status.sync.revision}}|{{(call .repo.GetCommitMetadata .app.status.sync.revision).Message}}> by {{(call .repo.GetCommitMetadata .app.status.sync.revision).Author}} in {{ .context.environment }}",
              } 
            }
          ]
        }]

which is an attachment like:

{
	"attachments": [
		{
			"blocks": [
				{
					"type": "section",
					"text": {
						"type": "mrkdwn",
						"text": "<https://github.com/repo/commit/sha|github> by author in dev:cookie:"
					}
				}
			]
		}
	]
}

and according to the slack block kit builder this is ok.
but in the argocd-notifications-controller I get this error:

level=error msg="Failed to notify recipient {slack argocd-notifications} defined in resource argo-cd/keycloak: failed to unmarshal attachments '[{\n  \"blocks\": [\n    {\n      \"type\": \"section\",\n      \"text\": {\n        \"type\": \"mrkdwn\",\n        \"text\": \"<https://github.com/repo/commit/sha|github> by author in dev:cookie:\",\n      } \n    }\n  ]\n}]\n' : invalid character '\\n' in string literal" resource=argo-cd/keycloak

In order to carry out the scenario of notification using cert-manager's CRD,
I made some changes to Getting Stated's ConfigMap and built it.
But the controller threw the following error and the notification to Slack did not work.
How should I deal with this error? The controller does not change the code, but uses the sample one.

cert-notifier nginx time="2021-12-28T09:19:03Z" level=error msg="Recovered from panic: runtime error: invalid memory address or nil pointer dereference
goroutine 64 [running]:
runtime/debug.Stack()
    /usr/local/go/src/runtime/debug/stack.go:24 +0x88
github.com/argoproj/notifications-engine/pkg/controller.(*notificationController).processQueueItem.func1(0x4000315860, {0x1073ce0, 0x40000990b0})
    /go/pkg/mod/github.com/argoproj/[email protected]/pkg/controller/controller.go:200 +0x48
panic({0x10e8be0, 0x20d7530})
  usr/local/go/src/runtime/panic.go:1038 +0x224
github.com/argoproj/notifications-engine/pkg/services.(*slackService).Send(0x40001d0af0, {{0x40000a19b0, 0x27}, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...)
    /go/pkg/mod/github.com/argoproj/[email protected]/pkg/services/slack.go:137 +0x91c
github.com/argoproj/notifications-engine/pkg/api.(*api).Send(0x4000283080, 0x40002b5650, {0x4000316e40, 0x1, 0x4}, {{0x40002c66f3, 0x5}, {0x4000040840, 0x14}})
    /go/pkg/mod/github.com/argoproj/[email protected]/pkg/api/api.go:71 +0x3f4
github.com/argoproj/notifications-engine/pkg/controller.(*notificationController).processResource(0x4000315860, {0x15ae0e8, 0x40004f4198}, 0x4000183dc0)
    /go/pkg/mod/github.com/argoproj/[email protected]/pkg/controller/controller.go:165 +0x708
github.com/argoproj/notifications-engine/pkg/controller.(*notificationController).processQueueItem(0x4000315860)
    /go/pkg/mod/github.com/argoproj/[email protected]/pkg/controller/controller.go:229 +0x38c
github.com/argoproj/notifications-engine/pkg/controller.(*notificationController).Run.func1()
    /go/pkg/mod/github.com/argoproj/[email protected]/pkg/controller/controller.go:118 +0x34
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x40000ebfc0)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x64
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x40000ebfc0, {0x1542240, 0x400012d020}, 0x1, 0x0)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0x94
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x40000ebfc0, 0x3b9aca00, 0x0, 0x1, 0x0)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x88
k8s.io/apimachinery/pkg/util/wait.Until(0x40000ebfc0, 0x3b9aca00, 0x0)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90 +0x48
created by github.com/argoproj/notifications-engine/pkg/controller.(*notificationController).Run
    go/pkg/mod/github.com/argoproj/[email protected]/pkg/controller/controller.go:117 +0x100"

config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: cert-manager-notifications-cm
data:
  trigger.on-cert-failed: |
    - when: any(cert.status.conditions, {.reason == 'DoesNotExist' && .status == 'False'})
      send: [cert-failed]

  template.cert-failed: |
    message: |
      Certificate {{.cert.metadata.name}} is failed!

  service.slack: |
    token: $slack-token