Giter VIP home page Giter VIP logo

monasca-monitor's Introduction

Monasca Monitor

https://travis-ci.org/stackhpc/monasca-monitor.svg?branch=master

A proof of concept system-level monitor for Monasca.

Performs a continuous system level test on a live Monasca deployment by perodically sending probe metrics to the Monasca API. The probe metrics are used to trigger an alarm, which fires a notification back to this service. The alarm notification events received by monasca-monitor are then made available to Prometheus to scrape. Prometheus can then be configured to alert a user if the notification events fail to arrive.

Instructions

Deploy with Ansible

The easiest way to install the Monasca monitor is to use the following Ansible role:

https://github.com/stackhpc/ansible-role-os-monasca-monitor

Manual install

First, create a Python virtualenv, for example:

$ virtualenv monasca-monitor
$ source monasca-monitor/bin/activate
$ python setup.py install

Create the configuration file. An example is provided in etc/monasca-monitor.conf.example. Typically this should be copied to the default config directory, but the path can also be specified with --config-file at run time. Remember to set the file system permissions correctly so that you reduce the risk of exposing your password.

$ mkdir -p /etc/monasca-monitor
$ cp etc/monasca-monitor.conf.example /etc/monasca-monitor/monasca-monitor.conf

Run the metrics source (it is recommended to do this in a screen session or similar):

$ mm-metric-source

Run the heart beat generator (this processes notifications from Monasca and makes them available to Prometheus):

$ mm-heartbeat

Make sure that the webhook notifier is enabled in the Monasca Notification service and then create a notification to ping this service back (assumes this service is running on the same host as the notification service):

$ monasca notification-create heartbeat_webhook webhook http://127.0.0.1:8000/heartbeat

Create an alarm to pick up the periodically sent metrics in Monasca:

$ monasca alarm-definition-create heartbeat 'sum(monascamonitor.heartbeat{}) >= 50' --description "Heartbeat" --severity LOW --alarm-actions 4f7f8448-5c47-4b92-914b-d9928f24e620

Configure Prometheus to scrape the endpoint (by default localhost:8000/metrics).

monasca-monitor's People

Contributors

dougszumski avatar wasaac avatar

Watchers

James Cloos avatar avatar Stig Telfer avatar

monasca-monitor's Issues

Retry mechanism required if Monasca API is temporarily unavailable 

Apr 12 08:59:53 mon1 mm-metric-source[6122]: Sending metric: {'jsonbody': [{'timestamp': 1555059593726, 'name': 'monascamonitor.heartbeat', 'dimensions': {'hostname': 'mon1'}, 'value': 0, 'value_meta': None}]}
Apr 12 08:59:53 mon1 mm-metric-source[6122]: Traceback (most recent call last):
Apr 12 08:59:53 mon1 mm-metric-source[6122]: File "/opt/monasca-monitor/bin/mm-metric-source", line 10, in <module>
Apr 12 08:59:53 mon1 mm-metric-source[6122]: sys.exit(main())
Apr 12 08:59:53 mon1 mm-metric-source[6122]: File "/opt/monasca-monitor/lib/python2.7/site-packages/monasca_monitor/metrics_source.py", line 106, in main
Apr 12 08:59:53 mon1 mm-metric-source[6122]: metricsource.send_metric()
Apr 12 08:59:53 mon1 mm-metric-source[6122]: File "/opt/monasca-monitor/lib/python2.7/site-packages/monasca_monitor/metrics_source.py", line 100, in send_metric
Apr 12 08:59:53 mon1 mm-metric-source[6122]: MetricSource._send_heartbeat_metric(self.monasca_client)
Apr 12 08:59:53 mon1 mm-metric-source[6122]: File "/opt/monasca-monitor/lib/python2.7/site-packages/monasca_monitor/metrics_source.py", line 97, in _send_heartbeat_metric
Apr 12 08:59:53 mon1 mm-metric-source[6122]: client.metrics.create(**kwargs)
Apr 12 08:59:53 mon1 mm-metric-source[6122]: File "/opt/monasca-monitor/lib/python2.7/site-packages/monascaclient/v2_0/metrics.py", line 31, in create
Apr 12 08:59:53 mon1 mm-metric-source[6122]: body = self.client.create(url=url_str, json=data)
Apr 12 08:59:53 mon1 mm-metric-source[6122]: File "/opt/monasca-monitor/lib/python2.7/site-packages/osc_lib/api/api.py", line 164, in create
Apr 12 08:59:53 mon1 mm-metric-source[6122]: ret = self._request(method, url, session=session, **params)
Apr 12 08:59:53 mon1 mm-metric-source[6122]: File "/opt/monasca-monitor/lib/python2.7/site-packages/osc_lib/api/api.py", line 141, in _request
Apr 12 08:59:53 mon1 mm-metric-source[6122]: return session.request(url, method, **kwargs)
Apr 12 08:59:53 mon1 mm-metric-source[6122]: File "/opt/monasca-monitor/lib/python2.7/site-packages/keystoneauth1/session.py", line 814, in request
Apr 12 08:59:53 mon1 mm-metric-source[6122]: resp = send(**kwargs)
Apr 12 08:59:53 mon1 mm-metric-source[6122]: File "/opt/monasca-monitor/lib/python2.7/site-packages/keystoneauth1/session.py", line 919, in _send_request
Apr 12 08:59:53 mon1 mm-metric-source[6122]: raise exceptions.ConnectFailure(msg)
Apr 12 08:59:53 mon1 mm-metric-source[6122]: keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://10.10.10.2:8082/v2.0/metrics: ('Connection aborted.', BadStatusLine("''",))
Apr 12 08:59:53 mon1 systemd[1]: monasca-monitor-metric-source.service: main process exited, code=exited, status=1/FAILURE
Apr 12 08:59:53 mon1 systemd[1]: Unit monasca-monitor-metric-source.service entered failed state.
Apr 12 08:59:53 mon1 systemd[1]: monasca-monitor-metric-source.service failed.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.