Giter VIP home page Giter VIP logo

pm2-health's Introduction

pm2-health

This PM2 module is:

  • Monitoring events (like app exit, restart etc.)
  • Monitoring app exceptions
  • Monitoring PMX metrics of your apps and send alerts when value hits treshold
  • Sending mail notifications

With rich config options you can fine-tune monitoring rules.

Installation & Upgrade

pm2 install pm2-health

Configuration

After installation run pm2 conf to configure module. Alternatively edit module_conf.json file directly (in PM2 home folder).

Minimum required config:

"pm2-health": {
    "smtp": {
        "host": "your-smtp-host",
        "port": 587,
        "from": "your-from-mail", // if not set, user will be used
        "user": "your-smtp-user",   // auth
        "password": "your-smtp-password",   // auth
        "secure": false,
        "disabled": false,
        "clientHostName": "your-machine.com", // optional, will force the client host-name FQDN used in SMTP HELLO. If not set, NodeMailer will ask host name to the OS, and use 127.0.0.1 if it's not a FQDN.
    },
    "mailTo": "mail1,mail2"
}

If any of required properties is not defined, pm2-health will shutdown. You can check error logs for details.

All config options:

  • smtp - SMTP server configuration. If your SMTP doesn't require auth, leave smtp.user empty

  • mailTo - comma separated list of notification receipients

  • replyTo - reply to address (optional)

  • events - list of events to monitor (optional). If not set, all events will be monitored.

Manually triggered events will not send notification.

  • exceptions - if true apps exceptions will be monitored (optional)

  • messages - if true apps custom messages will be monitored (optional). See Custom messages

  • messageExcludeExps - array of regular expressions used to exclude messages (optional). See Filtering custom messages

  • metric - object describing PMX metrics to be monitored (optional). See Metrics monitoring

  • metricIntervalS - how often PMX metrics will be tested in seconds (optional). If not set, 60 seconds is used

  • aliveTimeoutS - alive watchdog timeout interal in seconds. If not set watchdog function is off. See Process alive watchdog

  • addLogs - if true app logs will be added as mail attachement (optional)

  • appsExcluded - array of app names to exclude from monitoring (optional)

  • appsIncluded - array of app names to include, if set appsExcluded is ignored (optional)

  • webConfig - if set, some of the config settings can be downloaded from given url (optional). See Web config

  • debugLogEnabled - if true debug log is enabled, by default is false (optional)

  • batchPeriodM - enables message batching and sets batching period (optional). See Message batching

  • batchMaxMessages - max. messages in batch (optional). See Message batching

Metrics monitoring

pm2-health can monitor any PMX metrics defined in your apps.

To configure rules of alerting, setup metric section in module config file.

"metric": {
    "metric name": {
        "target": 0,
        "op": ">",
        "ifChanged": true,
        "noNotify": true,
        "noHistory": true,
        "exclude": false
    },
    "metric 2": {
        ...
    }    
}
  • metric name - name of metric defined in one of your apps

  • target - target numeric value

  • op - operator to compare metric value and target. Can be one of: <, >, =, <=, >=, !=

  • ifChanged - if true, alert will trigger only if current metric value is different from last recorded value (optional)

  • noNotify - if true, no alerts will be send (optional)

  • noHistory - if true, metric value history won't be stored (optional)

  • exclude - if true, metric will be complettely excluded from monitoring (optional)

  • direct - if true, metric value won't be converted to number (optional)

By default, cpu and memory metrics are added.

Learn how to define PMX metrics in your apps here: http://pm2.keymetrics.io/docs/usage/process-metrics/

Custom messages

On top of standard PM2 events, you can monitor custom messages sent from your apps.

To send message from your app use:

process.send({
    type: "process:msg",    
    data: {    
        ...
    }
});
  • type - must be process:msg

  • data - object containing additional data (optional).

Filtering custom messages

You can exclude some of the messages based on their data content:

  1. Add regular expression to list in messageExcludeExp config property
  2. data (converted to JSON string) will be tested with this all expressions in the list
  3. If any test is positive, message will be excluded

Example:

You wish to monitor slow operations in your app, so you send custom messages like so:

function slow(operation, duration) {
    process.send({ type: "process:msg", data: { operation, duration }});
}

You know that backup and restore operations are always slow and wish to exclude them, but still get other slow operations.

Set config to:

"messageExcludeExps": [
    "\"operation\": \"(backup|restore)\""
]

Remember to escape regex string to JSON string

Process alive watchdog

Alive watchdog (added in 1.9.0) can observe alive messages from processes.

To use functionallity your process has to send periodically process:msg signal as such:

process.send({
    type: "process:msg",    
    data: "alive"
});

In addition config parameter aliveTimeoutS must be added. If alive message won't be received within aliveTimeoutS (seconds), alert will be send.

aliveTimeoutS must be lower than interval of sending process:alive signal.

After first alert, following test is done every 10 minues for 6 consecutive times, after wich alerting stops, assuming process is permanetly closed.

Web config

Web config (added in 1.7) allows you to fetch some of the config settings from web url.

Sample config:

{
    "webConfig": {
        "url": "url of JSON file",
        "auth": {
            "user": "...",
            "password": "..."
        },
        "fetchIntervalM": 10
    }
}

Url must return UTF-8 JSON with config properties.

Only following properties can be used: events, metric, exceptions, messages, messageExcludeExps, appsExcluded, metricIntervalS, addLogs, batchPeriodM, batchMaxMessages

Message batching

Feature added in (1.11) allows to merge multiple messages over period of time and send them as single message.

This can be used to limit number of messages sent (prevent spam).

To enable please set following properties in config section:

  • batchPeriodM - period (in minutes) to batch, if not set batching is not enabled
  • batchMaxMessages - max. number of messages in batch (optional)

Batch message will be send after batchPeriodM elapses or if number of messages collected are greater than batchMaxMessages.

Priority messages (as exceptions etc.) are not batched and sent immediatelly.

It's advised to set batchMaxMessages to prevent huge messages.

Batching settings can be changed by web config.

Hold notifications temporarily

To hold mail notification: pm2 trigger pm2-health hold 30

Notifications will restart automatically after 30 minutes.

To unhold immediatelly: pm2 trigger pm2-health unheld

All monitoring processes continues, just mail notification is held

Mail template

Mail uses HTML format. To adjust template, you can edit Template.html

<!-- body --> will be exchanged with actual message body.

<!-- timeStamp --> will be exchanged with event timestamp (UTC).

pm2-health update will override your Template.html, so keep backup ๐Ÿ˜Š

Testing mail

To send test mail: pm2 trigger pm2-health mail

Building

pm2-health is written using TypeScript 2.6.1+ with es2017 target.

es2017 is supported by Node 8+. If you need to use ealier version, build solution using es5 or es6 target.

Solution includes VS Code settings for build and debug.

pm2-health's People

Contributors

dependabot[bot] avatar mgryniak avatar pankleks avatar tchatelin avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

pm2-health's Issues

Custom priority for exceptions and events

Hello, I really appreciate the work that you have made on this library, it is exactly what I was looking for.

The main reason for me to use this package is to message me, once the server is not able to start up after some restarts. The problem is that my e-mail inbox receives tons of e-mails with the same information value -> The application cannot boot up.

The possible solution would be to allow set custom priority for exceptions or at least for events. At this time, the events has high priority and thus bypasses the batchMaxMessages and batchPeriodM as mentioned in https://github.com/pankleks/pm2-health#message-batching.

Are you will to add this feature or at least can I do a PR for this?

Thanks.

Way to disable the log?

Hi,

Is there currently a way to disable internal logging for this module? By default it's written in .pm2/logs/pm2-health-out.log

Thanks

Disable auto-scaling

The module spawns multiple instances over time, which is usually not desired as I will receive multiple e-mails with the same information.

Is there any way to prevent that? I noticed that what I do fresh install of the plugin, it spawns only one instance (as expected), but when other apps are deployed/created/updated it spawns next instance.

Thanks.

Whitelist alternative to `appsExcluded`

Basically an appsIncluded option.
I plan to use this on a testing server, where most of the apps will be restarted a lot of times and I only want to monitor the status of a single worker which shouldn't restart

can't send mail -> getaddrinfo ENOTFOUND your-smtp-host

Hey Forum,

I've modified my /home/myuser/.pm2/module_conf.json to the following:

{
    "pm2-health": {
        "smtp": {
            "host": "mail.myserver.com",
            "port": 465,
            "user": "TOPSECRET",
            "password": "TOPSECRET",
            "from": "[email protected]",
            "secure": "true",
            "disabled": false
        },
        "mailTo": "[email protected]",
        "replyTo": "",
        "batchPeriodM": 0,
        "batchMaxMessages": 0,
        "events": [
            "exit,restart"
        ],
        "exceptions": true,
        "messages": true,
        "messageExcludeExps": [],
        "metric": {},
        "metricIntervalS": 60,
        "aliveTimeoutS": 300,
        "addLogs": true,
        "appsExcluded": [],
        "snapshot": {
            "url": "",
            "token": "",
            "auth": {
                "user": "",
                "password": ""
            },
            "disabled": false
        }
    },
    "module-db-v2": {
        "pm2-health": {}
    }
}

However, I'm unable to get it to work because: 0|pm2-heal | 2020-06-24T08:26:48.294Z: can't send mail -> getaddrinfo ENOTFOUND your-smtp-host

Any obvious mistake I made?

App is inactive

Every since I upgraded pm2-health to v1.11.8 pm2-health-out.log is being filled with the message "app is inactive". The message is correct because the server is just waiting for requests, but I'm getting way too many messages. I'm getting that message 3 times (at the same time) every minute, which is odd because I only have 1 instance of my app and pm2 -health running. It's making pm2-health's output in pm2 logs useless because it's just a bunch of copies of the same message.

Reading the documentation I don't see any obvious way to handle this problem. Thanks for any assistance you can provide.

SparkPost API Integration

Hello, I was wondering if pm2-health supports sparkpost integration. In order to use the sparkpost smtp server AUTH LOGIN authentication method is required. Is this currently supported? Thank you.

mail subject configuration option

It would be nice if it would be possible to alter the mail subject, all my scripts that send out mails have specific subjects based on the tool name, server name, location, ...

No emails are going out on test

Hi,

I can't make it to work, basically it everything installs, I added smtp.gmail.com and the rest of the stuff. When I trigger email test, it is not going out at all, not even hitting smtp.gmail.com..are there anything else needs to be done? I'm using node js 8.11.3, latest PM2 and everything is on Ubuntu.

How can I only monitor the specified id?

If i running app 0,1,2
and I wish to monitor only id 0, what should I do?
And
I want to send email only when there is a specific error in the log, such as connection timed out.

I try below But not useful.
Thankyou!

"metric": {
    "tg.line": {
        "target": 1,
        "op": ">=",
        "exclude": true
    }
}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.