fluent-plugin-notifier

Fluentd plugin to emit notifications for messages, with numbers over/under threshold, or specified pattern strings.

Requirements

fluent-plugin-notifier	fluentd	ruby
>= 1.0.0	>= v0.14.0	>= 2.1
< 1.0.0	< v0.14.0	>= 1.9

Configuration

To notify apache logs with over 1000000 (microseconds) duration for CRITICAL , or status '500' by string pattern match:

<match apache.log.**>
  @type notifier
  @label @notification_events
  <def>
    pattern apache_duration
    check numeric_upward
    warn_threshold  800000
    crit_threshold 1000000
    target_keys duration
  </def>
  <def>
    pattern status_500
    check string_find
    warn_regexp 5\d\d
    crit_regexp 500
    target_key_pattern ^status.*$
    exclude_key_pattern ^status_ignore_.*$  # key name not to notify about...
  </def>
</match>

With this configuration, you will get notification messages in <label @notification_events> section, like this:

2012-05-15 19:44:29 +0900 notification: {"pattern":"apache_duration","target_tag":"apache.log.xxx","target_key":"duration","check_type":"numeric_upward","level":"crit","threshold":1000000,"value":"1057231","message_time":"2012-05-15 19:44:27 +0900"}
2012-05-15 19:44:29 +0900 notification: {"pattern":"status_500","target_tag":"apache.log.xxx","target_key":"status","check_type":"string_find","level":"crit","regexp":"/500/","value":"500","message_time":"2012-05-15 19:44:27 +0900"}

Available 'check' types: 'numeric_upward', 'numeric_downward' and 'string_find'

Default configurations:

tag: 'notification'
- in top level, 'default_tag', 'default_tag_warn,' and 'default_tag_crit' available
- in each section, 'tag', 'tag_warn' and 'tag_crit' available
notification suppression
- at first, notified once in 1 minute, 5 times
- next, notified once in 5 minutes, 5 times
- last, notified once in 30 minutes
- in top level, 'default_interval_1st', 'default_interval_2nd', 'default_interval_3rd', 'default_repetitions_1st' and 'default_repetitions_2nd' available
- in each section, 'interval_1st', 'interval_2nd', 'interval_3rd', 'repetitions_1st' and 'repetitions_2nd' available

If you want to get every 5 minutes notifications (after 1 minutes notifications), specify '0' for 'repetitions_2nd'.

Message Testing

To include specified messages into check target, or to exclude specified messages from check target, directive is useful.

<match apache.log.**>
  @type notifier
  @label @notifications
  <test>
    check numeric
    target_key duration     # microseconds
    lower_threshold 5000    # 5ms
    upper_threshold 5000000 # 5s
  </test>
  <def>
    pattern status_500
    check string_find
    warn_regexp 5\d\d
    crit_regexp 500
    target_key_pattern ^status.*$
  </def>
</match>

<label @notifications>
  <match **>
    # send notifications to Slack, email or ...
  </match>
</label>

With configuration above, fluent-plugin-notifier checks messages with specified duration value (d: 5000 <= d <= 5000000), and others are ignored.

Available 'check' types are: 'numeric', 'regexp' and 'tag'.

numeric
- 'lower_threshold', 'upper_threshold' and both are available
regexp, tag
- 'include_pattern', 'exclude_pattern' and both are available
- 'tag' checks tag strings after 'input_tag_remove_prefix'

Multiple directives means logical AND of each tests.

<match apache.log.**>
  @type notifier
  @label @notifications
  input_tag_remove_prefix apache.log
  <test>
    check tag
    include_pattern ^news[123]$ # for specified web server log
  </test>
  <test>
    check numeric
    target_key duration     # microseconds
    lower_threshold 5000    # 5ms
  </test>
  <test>
    check regexp
    target_key vhost
    exclude_pattern ^image.news.example.com$  # ingore image delivery server log
  </test>
  <test>
    check regexp
    target_key path
    include_pattern ^/path/to/contents/    # for specified content path only
    exclude_pattern \.(gif|jpg|png|swf)$   # but image files are ignored
  </test>
  <def>
    pattern status_500
    check string_find
    warn_regexp 5\d\d
    crit_regexp 500
    target_key_pattern ^status.*$
  </def>
</match>

Notifier plugin configured like this will check messages:

with tag 'apache.log.news1', 'apache.log.news2' or 'apache.log.news3'
with duration bigger than 5ms (upper unlimited)
without vhost image.news.example.com
with request path '/path/to/contents/*' and without file suffix gif/jpg/png/swf.

TODO

patches welcome!

Copyright

License
- Apache License, Version 2.0

Having issue with configuration

Hello
I don't know if this plugin still works with the current fluentd version or if I am doing something wrong. Would be great if someone can help me out.

So my alert-email.conf looks like this:

<source>
  @type tail
  path /var/log/pods/.log # Set the location of your log file
  <parse>
    @type json
    time_key nil
  </parse>
  tag cm.log
</source>

<match cm.log>
  @type copy
  <store>
    @type stdout
  </store>
</match>

<match cm.log>
  @type notifier
  @label @notifications
  <def>
    pattern search_cmlog
    check string_find
    warn_regexp .*WARN.*
    crit_regexp .*INFO.*
    target_key_pattern .*WARN.*|.*INFO.*
  </def>
</match>

<label @notifications>
  <match **>
    @type copy
    <store>
      @type stdout  # Print to stdout for debugging
    </store>
    <store>
      @type mail
      host x        # Change this to your SMTP server host
      port x                    # Normally 25/587/465 are used for submission
      user x           # Use your username to log in
      password x              # Use your login password
      enable_starttls_auto true   # Use this option to enable STARTTLS
      from x         # Set the sender address
      to x         # Set the recipient address
      subject 'ERROR'
      out_keys target_tag, pattern, value, message_time
    </store>
  </match>
</label>

It works up until adding the @notifications label. For debug purposes I added the @copy stdout part.
In the log it looks like this:

2020-07-02 13:12:24.728443702 +0000 cm.log: {"log":"2020-07-02 15:12:24,309 WARN [.thread.FileSystemPollerThread] [--] Polling folder doesn't exist\n","stream":"stdout","time":"2020-07-02T13:12:24.309399034Z"}

So why does my regex and adding the new label not work?

Thanks in advance!

tagomoris / fluent-plugin-notifier Goto Github PK