Using Windows Server 2012 R2, rabbitmq 3.6.15, erlang OTP 20.2, with plugins from 3.6.14.1 release. Get the following error in console when enabling the plugin. Rabbitmq is installed as a service and the user which runs the service has full control of the plugin expand directory.

C:\Program Files (x86)\RabbitMQ Server\rabbitmq_server-3.6.15\sbin>rabbitmq-plugins.bat enable prometheus_process_collector
The following plugins have been enabled:
prometheus_process_collector

Applying plugin configuration to rabbit@MYQUEPP1... failed.
Error: {plugin_module_unloadable,"prometheus_process_collector",
{error,on_load_failure}}

I am unable to restart rabbitmq if the plugin is in enabled status. The following is generated in the rabbitmq.log:
=WARNING REPORT==== 27-Feb-2018::13:24:12 ===
The on_load function for module prometheus_process_collector returned:
{error,{load_failed,"Failed to load NIF library c:/Users/MYRAB~1/AppData/Roaming/RabbitMQ/db/rabbit@MYQUEPP1-plugins-expand/prometheus_process_collector-1.3.0/priv/prometheus_process_collector: 'The specified module could not be found.'"}}

=INFO REPORT==== 27-Feb-2018::13:24:12 ===
Error description:
{plugin_module_unloadable,"prometheus_process_collector",
{error,on_load_failure}}

Log files (may contain more information):
D:/Logs/[email protected]
D:/Logs/[email protected]

Stack trace:
[{rabbit_plugins,prepare_dir_plugin,1,
[{file,"src/rabbit_plugins.erl"},{line,249}]},
{rabbit_plugins,'-prepare_plugins/1-lc$^0/1-0-',2,
[{file,"src/rabbit_plugins.erl"},{line,213}]},
{rabbit_plugins,prepare_plugins,1,
[{file,"src/rabbit_plugins.erl"},{line,213}]},
{rabbit,broker_start,0,[{file,"src/rabbit.erl"},{line,300}]},
{rabbit,start_it,1,[{file,"src/rabbit.erl"},{line,424}]},
{init,start_em,1,[{file,"init.erl"},{line,1085}]},
{init,do_boot,3,[{file,"init.erl"},{line,793}]}]

As explained in #12 (comment) , the prometheus_process_collector plugin aborts the booting of RabbitMQ when the RABBITMQ_PLUGINS_EXPAND_DIR points to a directory which is mounted with noexec flag (or which is associated with a selinux policy which prevents executing code).

I found the issue trying to run RabbitMQ packaged as an Openshift3 (kubernetes) pod which persists its data to a persistent directory on the host. Since the Centos7 host has SELinux enabled by default, I had to chcon -Rt svirt_sandbox_file_t /data/rabbitmq to make the persistent directory available to the docker container. RabbitMQ booted fine until I modified my Dockerfile to enable the prometheus_process_collector; I then found out that the svirt_sandbox_file_t label on the directory only enables read and writes on the directory, enforcing the equivalent of the "noexec" mount flag. This made the loading of the native code in the plugin fail and aborted the boot.

Simple steps to reproduce:

$ sudo lvcreate -n rabbitmq -L 100M ssd
  Logical volume "rabbitmq" created.
$ sudo mkfs.xfs /dev/ssd/rabbitmq
<snip>
$ sudo mkdir -p /tmp/rabbitmq
$ sudo mount -o noexec,nodev,nosuid /dev/ssd/rabbitmq /tmp/rabbitmq
$ sudo chown 999:999 /tmp/rabbitmq # 999 is id of rabbitmq user in the official RabbitMQ image
$ docker run --rm -ti -u 999 -v /tmp/rabbitmq:/var/lib/rabbitmq -e RABBITMQ_BASE=/var/lib/rabbitmq deadtrickster/rabbitmq_prometheus:3.6.9.1


BOOT FAILED
===========

Error description:
   {plugin_module_unloadable,"prometheus_process_collector",
                             {error,on_load_failure}}

Log files (may contain more information):
   tty
   tty

Stack trace:
   [{rabbit_plugins,prepare_dir_plugin,1,
                    [{file,"src/rabbit_plugins.erl"},{line,241}]},
    {rabbit_plugins,'-prepare_plugins/1-lc$^1/1-1-',1,
                    [{file,"src/rabbit_plugins.erl"},{line,204}]},
    {rabbit_plugins,prepare_plugins,1,
                    [{file,"src/rabbit_plugins.erl"},{line,204}]},
    {rabbit,broker_start,0,[{file,"src/rabbit.erl"},{line,293}]},
    {rabbit,start_it,1,[{file,"src/rabbit.erl"},{line,417}]},
    {init,start_em,1,[]},
    {init,do_boot,3,[]}]


=WARNING REPORT==== 18-Apr-2017::18:08:46 ===
The on_load function for module prometheus_process_collector returned {error,
                                                                       {load_failed,
                                                                        "Failed to load NIF library: '/var/lib/rabbitmq/mnesia/rabbit@e4519f4d31e4-plugins-expand/prometheus_process_collector-1.0.2/priv/prometheus_process_collector.so: failed to map segment from shared object: Operation not permitted'"}}

=INFO REPORT==== 18-Apr-2017::18:08:46 ===
Error description:
   {plugin_module_unloadable,"prometheus_process_collector",
                             {error,on_load_failure}}

Log files (may contain more information):
   tty
   tty

Stack trace:
   [{rabbit_plugins,prepare_dir_plugin,1,
                    [{file,"src/rabbit_plugins.erl"},{line,241}]},
    {rabbit_plugins,'-prepare_plugins/1-lc$^1/1-1-',1,
                    [{file,"src/rabbit_plugins.erl"},{line,204}]},
    {rabbit_plugins,prepare_plugins,1,
                    [{file,"src/rabbit_plugins.erl"},{line,204}]},
    {rabbit,broker_start,0,[{file,"src/rabbit.erl"},{line,293}]},
    {rabbit,start_it,1,[{file,"src/rabbit.erl"},{line,417}]},
    {init,start_em,1,[]},
    {init,do_boot,3,[]}]

{"init terminating in do_boot",{plugin_module_unloadable,"prometheus_process_collector",{error,on_load_failure}}}
init terminating in do_boot ()

Crash dump is being written to: erl_crash.dump...%                                                                                                                                                                                $ 

Workaround: explictly set the RABBITMQ_PLUGINS_EXPAND_DIR environment variable to point to a path where code can be executed, for example RABBITMQ_PLUGINS_EXPAND_DIR=/tmp/rabbitmq-expand-plugins. Note that the parent directory of RABBITMQ_PLUGINS_EXPAND_DIR must be writable by the RabbitMQ user, because RabbitMQ will try to rm -rf ${RABBITMQ_PLUGINS_EXPAND_DIR} on boot.

In other words, run the container like this:

$ docker run --rm -ti -u 999 -v /tmp/rabbitmq:/var/lib/rabbitmq -e RABBITMQ_BASE=/var/lib/rabbitmq -e RABBITMQ_PLUGINS_EXPAND_DIR=/tmp/rabbitmq-plugins-expand deadtrickster/rabbitmq_prometheus:3.6.9.1

RabbitMQ has the ability to inform administrators when a connection (or channels, or queues) is in "flow control", meaning that the server can't process messages as fast as the connection wants to publish them. This can indicate all kinds of things, things that would be useful to be able to monitor.

It would be great if the exporter made this information available.

So, by default at /api/metrics/I see a very limited number of metrics, all describing the state of the system in VERY technical terms. Surely I can access metrics like message count per queue and such as well?

Looking at this I am pretty confident I should be able to do that, but I can't for the life of me figure out how!

Could you add an example or two to the readme?

I'm trying an installation of RabbitMQ 3.7 (3.7.1 and 3.7.2) on Debian Stretch (9.3) with Erlang 20.1.

The issue is that although I've copied all plugins to /usr/lib/rabbitmq/lib/rabbitmq_server-3.7.1/plugins, rabbitmq-plugins list shows only prometheus_rabbitmq_exporter, not the other plugins like in 3.6.x.

# rabbitmq-plugins list
Configured: E = explicitly enabled; e = implicitly enabled
| Status: * = running on rabbit@test-rabbitmq-37-puppet
|/
[E*] prometheus_rabbitmq_exporter      v3.7.1.1
[  ] rabbitmq_amqp1_0                  3.7.1
[  ] rabbitmq_auth_backend_cache       3.7.1
[  ] rabbitmq_auth_backend_http        3.7.1
[  ] rabbitmq_auth_backend_ldap        3.7.1
[  ] rabbitmq_auth_mechanism_ssl       3.7.1
[  ] rabbitmq_consistent_hash_exchange 3.7.1
[  ] rabbitmq_event_exchange           3.7.1
[  ] rabbitmq_federation               3.7.1
[  ] rabbitmq_federation_management    3.7.1
[  ] rabbitmq_jms_topic_exchange       3.7.1
[E*] rabbitmq_management               3.7.1
[e*] rabbitmq_management_agent         3.7.1
[  ] rabbitmq_mqtt                     3.7.1
[  ] rabbitmq_peer_discovery_aws       3.7.1
[  ] rabbitmq_peer_discovery_common    3.7.1
[  ] rabbitmq_peer_discovery_consul    3.7.1
[  ] rabbitmq_peer_discovery_etcd      3.7.1
[  ] rabbitmq_peer_discovery_k8s       3.7.1
[  ] rabbitmq_random_exchange          3.7.1
[  ] rabbitmq_recent_history_exchange  3.7.1
[  ] rabbitmq_sharding                 3.7.1
[  ] rabbitmq_shovel                   3.7.1
[  ] rabbitmq_shovel_management        3.7.1
[  ] rabbitmq_stomp                    3.7.1
[  ] rabbitmq_top                      3.7.1
[  ] rabbitmq_tracing                  3.7.1
[  ] rabbitmq_trust_store              3.7.1
[e*] rabbitmq_web_dispatch             3.7.1
[  ] rabbitmq_web_mqtt                 3.7.1
[  ] rabbitmq_web_mqtt_examples        3.7.1
[  ] rabbitmq_web_stomp                3.7.1
[  ] rabbitmq_web_stomp_examples       3.7.1

# l /usr/lib/rabbitmq/lib/rabbitmq_server-3.7.1/plugins
total 6124
-rw-r--r-- 1 root root   13395 Dec 21 22:27 accept-0.3.3.ez
-rw-r--r-- 1 root root  104602 Dec 21 13:32 amqp10_client-3.7.1.ez
-rw-r--r-- 1 root root   41194 Dec 21 13:32 amqp10_common-3.7.1.ez
-rw-r--r-- 1 root root  411074 Dec 21 13:32 amqp_client-3.7.1.ez
-rw-r--r-- 1 root root  224169 Dec 21 13:32 cowboy-2.0.0.ez
-rw-r--r-- 1 root root  438075 Dec 21 13:32 cowlib-2.0.0.ez
-rw-r--r-- 1 root root   69244 Dec 21 13:32 goldrush-0.1.9.ez
-rw-r--r-- 1 root root  101517 Dec 21 13:32 jsx-2.8.2.ez
-rw-r--r-- 1 root root  222981 Dec 21 13:32 lager-3.5.1.ez
-rw-r--r-- 1 root root  198993 Dec 21 22:28 prometheus-3.4.5.ez
-rw-r--r-- 1 root root   14344 Dec 21 22:28 prometheus_cowboy-0.1.4.ez
-rw-r--r-- 1 root root   22055 Dec 21 22:28 prometheus_httpd-2.1.8.ez
-rw-r--r-- 1 root root   17468 Dec 21 22:28 prometheus_process_collector-1.3.1.ez
-rw-r--r-- 1 root root  213774 Dec 21 22:28 prometheus_rabbitmq_exporter-v3.7.1.1.ez
-rw-r--r-- 1 root root  697498 Dec 21 13:32 rabbit_common-3.7.1.ez
-rw-r--r-- 1 root root  303410 Dec 21 13:32 rabbitmq_amqp1_0-3.7.1.ez
-rw-r--r-- 1 root root   31487 Dec 21 13:32 rabbitmq_auth_backend_cache-3.7.1.ez
-rw-r--r-- 1 root root   14430 Dec 21 13:32 rabbitmq_auth_backend_http-3.7.1.ez
-rw-r--r-- 1 root root   46977 Dec 21 13:32 rabbitmq_auth_backend_ldap-3.7.1.ez
-rw-r--r-- 1 root root   20115 Dec 21 13:32 rabbitmq_auth_mechanism_ssl-3.7.1.ez
-rw-r--r-- 1 root root   67586 Dec 21 13:32 rabbitmq_aws-3.7.1.ez
-rw-r--r-- 1 root root   19055 Dec 21 13:32 rabbitmq_consistent_hash_exchange-3.7.1.ez
-rw-r--r-- 1 root root   16027 Dec 21 13:32 rabbitmq_event_exchange-3.7.1.ez
-rw-r--r-- 1 root root  236902 Dec 21 13:32 rabbitmq_federation-3.7.1.ez
-rw-r--r-- 1 root root   14430 Dec 21 13:32 rabbitmq_federation_management-3.7.1.ez
-rw-r--r-- 1 root root   27029 Dec 21 13:32 rabbitmq_jms_topic_exchange-3.7.1.ez
-rw-r--r-- 1 root root 1034453 Dec 21 13:32 rabbitmq_management-3.7.1.ez
-rw-r--r-- 1 root root  178890 Dec 21 13:32 rabbitmq_management_agent-3.7.1.ez
-rw-r--r-- 1 root root  156746 Dec 21 13:32 rabbitmq_mqtt-3.7.1.ez
-rw-r--r-- 1 root root   25611 Dec 21 13:32 rabbitmq_peer_discovery_aws-3.7.1.ez
-rw-r--r-- 1 root root   54697 Dec 21 13:32 rabbitmq_peer_discovery_common-3.7.1.ez
-rw-r--r-- 1 root root   50329 Dec 21 13:32 rabbitmq_peer_discovery_consul-3.7.1.ez
-rw-r--r-- 1 root root   33527 Dec 21 13:32 rabbitmq_peer_discovery_etcd-3.7.1.ez
-rw-r--r-- 1 root root   14960 Dec 21 13:32 rabbitmq_peer_discovery_k8s-3.7.1.ez
-rw-r--r-- 1 root root    7689 Dec 21 13:32 rabbitmq_random_exchange-3.7.1.ez
-rw-r--r-- 1 root root   19126 Dec 21 13:32 rabbitmq_recent_history_exchange-3.7.1.ez
-rw-r--r-- 1 root root   49752 Dec 21 13:32 rabbitmq_sharding-3.7.1.ez
-rw-r--r-- 1 root root  177887 Dec 21 13:32 rabbitmq_shovel-3.7.1.ez
-rw-r--r-- 1 root root   23971 Dec 21 13:32 rabbitmq_shovel_management-3.7.1.ez
-rw-r--r-- 1 root root  146079 Dec 21 13:32 rabbitmq_stomp-3.7.1.ez
-rw-r--r-- 1 root root   71960 Dec 21 13:32 rabbitmq_top-3.7.1.ez
-rw-r--r-- 1 root root   61791 Dec 21 13:32 rabbitmq_tracing-3.7.1.ez
-rw-r--r-- 1 root root   71545 Dec 21 13:32 rabbitmq_trust_store-3.7.1.ez
-rw-r--r-- 1 root root   44551 Dec 21 13:32 rabbitmq_web_dispatch-3.7.1.ez
-rw-r--r-- 1 root root   38507 Dec 21 13:32 rabbitmq_web_mqtt-3.7.1.ez
-rw-r--r-- 1 root root   66307 Dec 21 13:32 rabbitmq_web_mqtt_examples-3.7.1.ez
-rw-r--r-- 1 root root   52653 Dec 21 13:32 rabbitmq_web_stomp-3.7.1.ez
-rw-r--r-- 1 root root   51995 Dec 21 13:32 rabbitmq_web_stomp_examples-3.7.1.ez
-rw-r--r-- 1 root root   56786 Dec 21 13:32 ranch-1.4.0.ez
-rw-r--r-- 1 root root   35381 Dec 21 13:32 ranch_proxy_protocol-1.4.4.ez
-rw-r--r-- 1 root root      59 Dec 21 13:32 README
-rw-r--r-- 1 root root   55901 Dec 21 13:32 recon-2.3.2.ez

Hi,

Would you be willing to re-compile all plugins against R16B03?

RabbitMQ 3.6.x supports R16B03 onwards, so I think that it would make sense for plugins to support the same version as well.

I'm running prometheus_process_collector on 19.3.6.2 and I get Error: {plugin_built_with_incompatible_erlang,"prometheus_process_collector"} when enabling the plugin.

Thanks!

I am trying to use the 3.6.12 release of this plugin with the 3.6.12-1 release of RabbitMQ on top of Ubuntu 16.04. Using the Ubuntu packages provided at rabbitmq.com. Erlang version is 18.3.

The required plugins are installed and enabled (rabbitmq-plugins enable --offline accept prometheus prometheus_httpd prometheus_rabbitmq_exporter).

The start of rabbitmq-server failed with the following entry in the logs:

=ERROR REPORT==== 7-Nov-2017::11:31:29 ===
Loading of /var/lib/rabbitmq/mnesia/rabbit-plugins-expand/prometheus-3.4.0/ebin/prometheus_sup.beam failed: badfile

=ERROR REPORT==== 7-Nov-2017::11:31:29 ===
beam/beam_load.c(1189): Error loading module prometheus_sup:
  mandatory chunk of type 'Atom' not found



=INFO REPORT==== 7-Nov-2017::11:31:29 ===
Error description:
   {could_not_start,prometheus,
       {undef,
           [{prometheus_sup,start_link,[],[]},
            {application_master,start_it_old,4,
                [{file,"application_master.erl"},{line,273}]}]}}

Hi @deadtrickster

What is the default TCP port the /api/metrics endpoint is exposed to? Is this somehow configurable as well?

Best regards,
Bart

I packaged this plugin as a part of my custom RabbitMQ docker image, which is based on the official rabbitmq:3.6 image.

Until today, the plugin worked (RabbitMQ version was 3.6.6). I rebuilt the image today, RabbitMQ was upgraded to version 3.6.8, and this plugin does not work anymore. When I curl the /api/metrics route on the management port I get 404 error. As a workaround, I rebuilt my image using rabbitmq:3.6.6 as the base, but I still want to report the issue to get it fixed.

Here are the list of the plugins installed in my image:

=INFO REPORT==== 24-Mar-2017::17:15:37 ===
Server startup complete; 11 plugins started.
 * prometheus_rabbitmq_exporter
 * rabbitmq_management
 * rabbitmq_web_dispatch
 * autocluster
 * rabbitmq_delayed_message_exchange
 * rabbitmq_management_agent
 * prometheus_process_collector
 * amqp_client
 * cowboy
 * cowlib
 * prometheus
=INFO REPORT==== 24-Mar-2017::17:17:18 ===

Hello, I would like to filter out all queue metric. And keep just some basic cluster health etc..

I'm expecting:

• faster scrape time,
• reducing cpu usage while scraping
• and no filtering rules in the Prometheus configuration

With basic configuration (no filtering) i get 85364 rows! (scrape duration ~10sec)
With the following config i got 51801 rows! (scrape duration ~8sec) I'ts still too much...

 {prometheus, [
   {rabbitmq_exporter, [
     {path, "/metrics"},
     {connections_total_enabled, true},
     {memory_stat_enabled,false},
     {queue_messages_stat, [{messages_published_total,false},
                           {messages_confirmed_total,false},
                           {messages_delivered_total,false},
                           {messages_delivered_no_ack_total,false},
                           {messages_get_total,false},
                           {messages_get_no_ack_total,false},
                           {messages_deliver_get_total,false},
                           {messages_redelivered_total,false},
                           {messages_returned_total,false}]},
     {exchange_messages_stat, [{messages_published_total,false},
                           {messages_published_in_total,false},
                           {messages_published_out_total,false},
                           {messages_confirmed_total,false},
                           {messages_delivered_total,false},
                           {messages_delivered_no_ack_total,false},
                           {messages_get_total,false},
                           {messages_get_no_ack_total,false},
                           {messages_deliver_get_total,false},
                           {messages_redelivered_total,false},
                           {messages_returned_total,false}]}
   ]}

Metric counts summary:

awk -F{ '{print $1}' /rb-plugin_SHORT | grep -v '^#' | sort | uniq -c | sort -h
      1
      1 erlang_mnesia_committed_transactions 4713
      1 erlang_mnesia_failed_transactions 2
      1 erlang_mnesia_held_locks 0
      1 erlang_mnesia_lock_queue 0
      1 erlang_mnesia_logged_transactions 5
      1 erlang_mnesia_restarted_transactions 765
      1 erlang_mnesia_transaction_coordinators 0
      1 erlang_mnesia_transaction_participants 0
      1 erlang_vm_atom_count 38404
      1 erlang_vm_atom_limit 5000000
      1 erlang_vm_dirty_cpu_schedulers 24
      1 erlang_vm_dirty_cpu_schedulers_online 24
      1 erlang_vm_dirty_io_schedulers 10
      1 erlang_vm_ets_limit 50000
      1 erlang_vm_logical_processors 24
      1 erlang_vm_logical_processors_available 24
      1 erlang_vm_logical_processors_online 24
      1 erlang_vm_memory_dets_tables 1
      1 erlang_vm_memory_ets_tables 177
      1 erlang_vm_port_count 18
      1 erlang_vm_port_limit 65536
      1 erlang_vm_process_count 7446
      1 erlang_vm_process_limit 1048576
      1 erlang_vm_schedulers 24
      1 erlang_vm_schedulers_online 24
      1 erlang_vm_smp_support 1
      1 erlang_vm_statistics_bytes_output_total 82095010
      1 erlang_vm_statistics_bytes_received_total 89728826
      1 erlang_vm_statistics_context_switches 6522008
      1 erlang_vm_statistics_garbage_collection_bytes_reclaimed 12941869408
      1 erlang_vm_statistics_garbage_collection_number_of_gcs 5478847
      1 erlang_vm_statistics_garbage_collection_words_reclaimed 1617733676
      1 erlang_vm_statistics_reductions_total 1374252069
      1 erlang_vm_statistics_run_queues_length_total 0
      1 erlang_vm_statistics_runtime_milliseconds 543432
      1 erlang_vm_statistics_wallclock_time_milliseconds 573721
      1 erlang_vm_thread_pool_size 384
      1 erlang_vm_threads 1
      1 erlang_vm_time_correction 1
      1 rabbitmq_channels 3813
      1 rabbitmq_connections 3813
      1 rabbitmq_connections_total
      1 rabbitmq_consumers 3975
      1 rabbitmq_exchanges 1143
      1 rabbitmq_messages_confirmed_total 1742937
      1 rabbitmq_messages_delivered_no_ack_total 0
      1 rabbitmq_messages_delivered_total 1588108
      1 rabbitmq_messages_deliver_get_total 1588108
      1 rabbitmq_messages_get_no_ack_total 0
      1 rabbitmq_messages_get_total 0
      1 rabbitmq_messages_published_in_total NaN
      1 rabbitmq_messages_published_out_total NaN
      1 rabbitmq_messages_published_total 1743471
      1 rabbitmq_messages_ready 0
      1 rabbitmq_messages_redelivered_total 197
      1 rabbitmq_messages_returned_total 0
      1 rabbitmq_messages_unacknowledged 0
      1 rabbitmq_queues 2332
      1 rabbitmq_queues_disk_reads 0
      1 rabbitmq_queues_disk_writes 0
      1 telemetry_scrape_duration_seconds_count
      1 telemetry_scrape_duration_seconds_sum
      1 telemetry_scrape_size_bytes_count
      1 telemetry_scrape_size_bytes_sum
      2 erlang_vm_memory_atom_bytes_total
      2 erlang_vm_memory_bytes_total
      2 erlang_vm_memory_processes_bytes_total
      2 telemetry_scrape_encoded_size_bytes_count
      2 telemetry_scrape_encoded_size_bytes_sum
      5 erlang_vm_memory_system_bytes_total
      5 rabbitmq_node_up
     17 rabbitmq_mnesia_table_disc_copies
     17 rabbitmq_mnesia_table_disc_only_copies
     17 rabbitmq_mnesia_table_disk_size_bytes
     17 rabbitmq_mnesia_table_local_content
     17 rabbitmq_mnesia_table_majority_required
     17 rabbitmq_mnesia_table_master_nodes
     17 rabbitmq_mnesia_table_memory_bytes
     17 rabbitmq_mnesia_table_ram_copies
     17 rabbitmq_mnesia_table_read_only
     17 rabbitmq_mnesia_table_records_count
   2332 rabbitmq_queue_auto_delete
   2332 rabbitmq_queue_consumers
   2332 rabbitmq_queue_consumer_utilisation
   2332 rabbitmq_queue_disk_size_bytes
   2332 rabbitmq_queue_durable
   2332 rabbitmq_queue_exclusive
   2332 rabbitmq_queue_head_message_timestamp
   2332 rabbitmq_queue_memory
   2332 rabbitmq_queue_message_bytes
   2332 rabbitmq_queue_message_bytes_persistent
   2332 rabbitmq_queue_message_bytes_ram
   2332 rabbitmq_queue_message_bytes_ready
   2332 rabbitmq_queue_message_bytes_unacknowledged
   2332 rabbitmq_queue_messages
   2332 rabbitmq_queue_messages_persistent
   2332 rabbitmq_queue_messages_ram
   2332 rabbitmq_queue_messages_ready
   2332 rabbitmq_queue_messages_ready_ram
   2332 rabbitmq_queue_messages_unacknowledged
   2332 rabbitmq_queue_messages_unacknowledged_ram
   2332 rabbitmq_queue_state
   2576 erlang_vm_allocators

Starting with 3.7.x plugins must depend on the rabbit app (see Breaking Changes: apparently I cannot link to a section, sorry) or else they won't appear in rabbitmq-plugins list output.

This is not an issue if the plugin is "pre-enabled" (was enabled previously or the node uses a generated enabled_plugins file) but it is a major problem for beginners.

Is there a metric per queue available whether the queue is responsive or not (e.g. rabbitmq_queue_unresponsive true|false)?

If not, can we add one?

There are several queueinfoitems that are currently not available as metric. https://www.rabbitmq.com/man/rabbitmqctl.1.man.html

Personally I was specifically looking for head_message_timestamp. Monitoring this timestamp enables me to verify there is no excessive delay in the system.
This items returns a timestamp, but for comparison in prometheus, it would be way easier if the return value was the same as time() (returns the number of seconds since January 1, 1970 UTC).

I have enabled prometheus_process_collector plugin, but can't see any related metrics:

$ rabbitmq-plugins list | grep prometheus
[E*] prometheus                        3.4.3
[E*] prometheus_httpd                  2.1.8
[E*] prometheus_process_collector      1.3.0
[E*] prometheus_rabbitmq_exporter      v3.6.14.1

$ curl -sS 127.0.0.1:15672/api/metrics | grep HELP | grep process_
# HELP erlang_vm_process_count The number of processes currently existing at the local node.
# HELP erlang_vm_process_limit The maximum number of simultaneously existing processes at the local node.

Can someone advice me what i'm doing wrong?

I am in the process of setting up an Ubuntu Bionic (18.04) RabbitMQ cluster, and am trying to get this exporter working. The APT repos for this distro have RabbitMQ 3.6.x instead of 3.7.x, and it may be why it's showing me an error when enabling prometheus_cowboy.

The releases page for the 3.6.x branch of this plugin don't have a binary for prometheus_cowboy. I can see cowboy is already an enabled plugin in the default RabbitMQ install.

Should I just remove the rabbitmq-plugins enable --offline prometheus_cowboy from my setup?

Summary

If a queue is created after the prometheus plugin has initialized, then subsequent called to /api/metrics result in "500 internal error". Restarting rabbitmq after still results in "500 internal error" every call to /api/metrics.

I encountered the bug with rabbitmq 3.6.8 and 3.6.9, no idea if it applies to earlier versions.

I am using the *.ez files from the latest stable release : https://github.com/deadtrickster/prometheus_rabbitmq_exporter/releases/tag/rabbitmq-3.6.8.1

The nasty stack trace looks like this:

=SUPERVISOR REPORT==== 12-Apr-2017::13:52:09 ===
     Supervisor: {local,rabbit_mgmt_db_cache_sup}
     Context:    child_terminated
     Reason:     {badarg,
                     [{dict,fetch,
                          [{resource,unknown,queue,
                               {resource,<<"/">>,queue,<<"boom">>}},
                           {dict,0,16,16,8,80,48,
                               {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                []},
                               {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                 []}}}],
                          [{file,"dict.erl"},{line,131}]},
                      {rabbit_mgmt_db,'-list_queue_stats/3-lc$^1/1-1-',4,
                          [{file,"src/rabbit_mgmt_db.erl"},{line,363}]},
                      {rabbit_mgmt_db,list_queue_stats,3,
                          [{file,"src/rabbit_mgmt_db.erl"},{line,360}]},
                      {timer,tc,2,[{file,"timer.erl"},{line,181}]},
                      {rabbit_mgmt_db_cache,handle_call,3,
                          [{file,"src/rabbit_mgmt_db_cache.erl"},{line,107}]},
                      {gen_server,try_handle_call,4,
                          [{file,"gen_server.erl"},{line,615}]},
                      {gen_server,handle_msg,5,
                          [{file,"gen_server.erl"},{line,647}]},
                      {proc_lib,init_p_do_apply,3,
                          [{file,"proc_lib.erl"},{line,247}]}]}
     Offender:   [{pid,<0.613.0>},
                  {id,rabbit_mgmt_db_cache_queues},
                  {mfargs,{rabbit_mgmt_db_cache,start_link,[queues]}},
                  {restart_type,permanent},
                  {shutdown,5000},
                  {child_type,worker}]

How to reproduce

In one terminal, run:

$ git clone https://github.com/deadtrickster/prometheus_rabbitmq_exporter.git
$ docker build -t dev-rabbitmq .
$ docker run --rm -ti -p 15672:15672 dev-rabbitmq

In your web browser:

• open http://localhost:15672/api/metrics
• notice that metrics are returned
• open http://localhost:15672/#/queues
• unfold the "Add a new queue" link
• create a queue named "boom" with default attributes (Durable, not AutoDelete)
• open http://localhost:15672/api/metrics
• the browser displays an empty page, http status is 500
• the docker logs displays nasty stack traces like the one above

when enabling rabbitmq_management_metrics on rabbitmq 3.6.2 the management webinterface complains about undefined: There is no template at js/tmpl/undefined.ejs

in the firefox dev tools I can see, that some where in the request a ressource "undefined" is requested and not found.

those are the plugins present on this system:

 Configured: E = explicitly enabled; e = implicitly enabled
 | Status:   * = running on rabbit@levigo-rmq
 |/
[e*] amqp_client                       3.6.2
[  ] cowboy                            1.0.3
[  ] cowlib                            1.0.1
[e*] mochiweb                          2.13.1
[  ] prometheus                        0.1.1
[  ] rabbitmq_amqp1_0                  3.6.2
[  ] rabbitmq_auth_backend_ldap        3.6.2
[  ] rabbitmq_auth_mechanism_ssl       3.6.2
[  ] rabbitmq_consistent_hash_exchange 3.6.2
[  ] rabbitmq_event_exchange           3.6.2
[  ] rabbitmq_federation               3.6.2
[  ] rabbitmq_federation_management    3.6.2
[E*] rabbitmq_management               3.6.2
[e*] rabbitmq_management_agent         3.6.2
[  ] rabbitmq_management_metrics       3.6.2
[  ] rabbitmq_management_visualiser    3.6.2
[  ] rabbitmq_mqtt                     3.6.2
[  ] rabbitmq_recent_history_exchange  1.2.1
[  ] rabbitmq_sharding                 0.1.0
[  ] rabbitmq_shovel                   3.6.2
[  ] rabbitmq_shovel_management        3.6.2
[  ] rabbitmq_stomp                    3.6.2
[  ] rabbitmq_tracing                  3.6.2
[e*] rabbitmq_web_dispatch             3.6.2
[  ] rabbitmq_web_stomp                3.6.2
[  ] rabbitmq_web_stomp_examples       3.6.2
[  ] sockjs                            0.3.4
[e*] webmachine                        1.10.3

the bug does not appear when enabling prometheus only - but it does as soon as I enable rabbitmq_management_metrics. And it does so no matter if I have prometheus enabled or disabled.

Suggest adding label self="1" to identify metrics that belong to the node performing the current scrape.

Reason: Greatly simplify identification of authoritative metrics.

Example: 3-node cluster, each cluster-wide metric is returned 3*3=9 times. Only 3 of them are from a "self" node, the other 6 are "last-reported" values, and are not reliable under high load.

We currently introduce a synthetic label (nodenum) in the query to perform a join on the authoritative node name. Each node has a unique numeric suffix:
label_replace(irate(rabbitmq_connection_send_bytes{environment="$environment",nodetype="$nodetype"}[2m]), "nodenum", "$1", "instance", ".?(\d+)")
AND
label_replace(irate(rabbitmq_connection_send_bytes{environment="$environment",nodetype="$nodetype"}[2m]), "nodenum", "$1", "node", ".?(\d+)")

Hi All,
images used ->
rabbitmq:3.7.8-management-alpine
prom/prometheus:v2.3.2
stefanprodan/swarmprom-grafana:5.0.1

I have used this https://grafana.com/dashboards/4279 json file for rabbitmq but metrics are not coming

Below are the docker logs

root@ip-10-85-0-239:~/Build_rabbitmq# docker logs ae4bd2e788346dc643602b38524148258447dcb4463c0ba2d77c91d0709ea4c9
WARNING: Removing trailing slash from RABBITMQ_PLUGINS_EXPAND_DIR
WARNING: Removing trailing slash from RABBITMQ_PLUGINS_EXPAND_DIR
2018-10-31 09:42:33.668 [info] <0.33.0> Application lager started on node rabbit@ae4bd2e78834
2018-10-31 09:42:33.738 [error] <0.5.0>
Error description:
    init:do_boot/3
    init:start_em/1
    rabbit:start_it/1 line 461
    rabbit:broker_start/0 line 337
    rabbit:start_apps/2 line 519
    app_utils:load_applications([os_mon,mnesia,rabbit_common,rabbit])
error:undef
Log file(s) (may contain more information):
   <stdout>

BOOT FAILED
===========

Error description:
    init:do_boot/3
    init:start_em/1
    rabbit:start_it/1 line 461
    rabbit:broker_start/0 line 337
    rabbit:start_apps/2 line 519
    app_utils:load_applications([os_mon,mnesia,rabbit_common,rabbit])
error:undef
Log file(s) (may contain more information):
   <stdout>

init terminating in do_boot (undef)
{"init terminating in do_boot",undef}

Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done

Please let me how to fix this issue.

Thank you in advance.

I'm trying to install the plugin in a rabbitmq 3.6.1

After enabling the plugins the gui complains about undefined: There is no template at js/tmpl/undefined.ejs. How to fix this?

I'm noticing locally I'm able to hit the /api/metrics endpoint without being authenticated to the RabbitMQ management portal.

Is there a way to use the basic auth for this endpoint that the RabbitMQ management plugin provides?

Add RabbitMQ specific tables stat using http://erlang.org/doc/man/mnesia.html#table_info-2.

Metric template rabbitmq_mnesia_table_<table_info_key>{table="<table_name>"}.

Tables used by RabbitMQ:

• rabbit_durable_exchange;
• rabbit_durable_queue;
• rabbit_durable_route;
• rabbit_exchange;
• rabbit_exchange_serial;
• rabbit_listener;
• rabbit_queue;
• rabbit_reverse_route;
• rabbit_route;
• rabbit_runtime_parameters;
• rabbit_semi_durable_route;
• rabbit_topic_trie_binding;
• rabbit_topic_trie_edge;
• rabbit_topic_trie_node;
• rabbit_user;
• rabbit_user_permission;
• rabbit_vhost.

Wait for deadtrickster/prometheus.erl#38

I followed the install to the letter (along with the dependencies). I am running Rabbit 3.7.8 on Linux is there a version dependency of Rabbit? Am I on too new of a version?

Hi,
When I use this Dockerfile, I get an error message saying
"Permisson denied: /etc/rabbitmq/rabbitmq.config"

any help is greatly appreciated

Hi,

I have a cluster of 3 rabbitmq instances. When using prometheus_rabbitmq_exporter version v3.6.5.8 I get the following error on prometheus server side:

text format parsing error in line 147: second TYPE line for metric name "rabbitmq_nodes_up", or TYPE reported after samples

The problem seems to be this:

# TYPE rabbitmq_nodes_up untyped
# HELP rabbitmq_nodes_up Node runnning status
rabbitmq_nodes_up{name="[email protected]",type="disc"} 1
# TYPE rabbitmq_nodes_up untyped
# HELP rabbitmq_nodes_up Node runnning status
rabbitmq_nodes_up{name="[email protected]",type="disc"} 1
# TYPE rabbitmq_nodes_up untyped
# HELP rabbitmq_nodes_up Node runnning status
rabbitmq_nodes_up{name="[email protected]",type="disc"} 1

while there are 3 TYPE lines for the same metric name, which is in this case rabbitmq_nodes_up.

Similar problem described here:

https://groups.google.com/forum/#!topic/prometheus-developers/ZVb2NjjUJ5A

Anyways, it seems that it should be printed only once:

# TYPE rabbitmq_nodes_up untyped
# HELP rabbitmq_nodes_up Node runnning status

instead of 3 times.

Thanks and Greets!

memory & disk, file descriptors & sockets are really important parameter to be watched/monitored for rabbitmq and it will be much helpful to add these into prometheus metrics. Use case include using these metrics in alert manager to trigger the alarm accordingly. this is more like an enhancement.

Best,

Hi,

As far as I understand, when running clustered RabbitMQ, by scraping individual node will return mostly same metrics for those that start with rabbitmq_. I guess this is due to stats are shared between nodes.

How would you recommend to scrape this? How will I know which shared which are individual per node?

For example:

• rabbitmq_consumers are per node
• rabbitmq_exchange_messages_published_out_total are shared

Thanks

I am building a custom rabbitmq docker images (based on the official rabbitmq 3.6.12 image) that includes this collector. The image builds successfully but running it fails with the following message:

Failed to enable plugin "prometheus_process_collector": it may have been built with an incompatible (more recent?) version of Erlang

Would it be possible to release new binaries of this plugin that are compatible with the current version of the official rabbitmq:3.6.12 docker image? Thank you very much.

Here is the Dockerfile I used:

$ cat Dockerfile
FROM rabbitmq:3.6.12

# prometheus exporter plugin
ADD ["https://github.com/deadtrickster/prometheus_rabbitmq_exporter/releases/download/rabbitmq-3.6.12.1/accept-0.3.3.ez", \
     "https://github.com/deadtrickster/prometheus_rabbitmq_exporter/releases/download/rabbitmq-3.6.12.1/prometheus-3.4.0.ez", \
     "https://github.com/deadtrickster/prometheus_rabbitmq_exporter/releases/download/rabbitmq-3.6.12.1/prometheus_httpd-2.1.4.ez", \
     "https://github.com/deadtrickster/prometheus_rabbitmq_exporter/releases/download/rabbitmq-3.6.12.1/prometheus_process_collector-1.1.0.ez", \
     "https://github.com/deadtrickster/prometheus_rabbitmq_exporter/releases/download/rabbitmq-3.6.12.1/prometheus_rabbitmq_exporter-v3.6.12.1.ez", \
     "/usr/lib/rabbitmq/lib/rabbitmq_server-${RABBITMQ_VERSION}/plugins/"]
RUN chmod a+r /usr/lib/rabbitmq/lib/rabbitmq_server-${RABBITMQ_VERSION}/plugins/prometheus*.ez /usr/lib/rabbitmq/lib/rabbitmq_server-${RABBITMQ_VERSION}/plugins/accept*.ez \
    && rabbitmq-plugins enable --offline accept prometheus_rabbitmq_exporter prometheus_process_collector prometheus_httpd \
    && chmod -R 777 /etc/rabbitmq
$ docker build -t myrabbitmq .
$ docker run --rm -ti myrabbitmq

=ERROR REPORT==== 3-Oct-2017::10:00:34 ===
Loading of /var/lib/rabbitmq/mnesia/rabbit@99e8e5aaa7e3-plugins-expand/prometheus_process_collector-1.1.0/ebin/prometheus_process_collector.beam failed: badfile

=ERROR REPORT==== 3-Oct-2017::10:00:34 ===
beam/beam_load.c(1287): Error loading module prometheus_process_collector:
  mandatory chunk of type 'Atom' not found




BOOT FAILED
===========

Error description:
   {plugin_built_with_incompatible_erlang,"prometheus_process_collector"}

Log files (may contain more information):
   tty
   tty

Stack trace:
   [{rabbit_plugins,prepare_dir_plugin,1,
                    [{file,"src/rabbit_plugins.erl"},{line,236}]},
    {rabbit_plugins,'-prepare_plugins/1-lc$^0/1-0-',2,
                    [{file,"src/rabbit_plugins.erl"},{line,202}]},
    {rabbit_plugins,prepare_plugins,1,
                    [{file,"src/rabbit_plugins.erl"},{line,202}]},
    {rabbit,broker_start,0,[{file,"src/rabbit.erl"},{line,293}]},
    {rabbit,start_it,1,[{file,"src/rabbit.erl"},{line,417}]},
    {init,start_em,1,[]},
    {init,do_boot,3,[]}]


=ERROR REPORT==== 3-Oct-2017::10:00:34 ===
Failed to enable plugin "prometheus_process_collector": it may have been built with an incompatible (more recent?) version of Erlang

=INFO REPORT==== 3-Oct-2017::10:00:34 ===
Error description:
   {plugin_built_with_incompatible_erlang,"prometheus_process_collector"}

Log files (may contain more information):
   tty
   tty

Stack trace:
   [{rabbit_plugins,prepare_dir_plugin,1,
                    [{file,"src/rabbit_plugins.erl"},{line,236}]},
    {rabbit_plugins,'-prepare_plugins/1-lc$^0/1-0-',2,
                    [{file,"src/rabbit_plugins.erl"},{line,202}]},
    {rabbit_plugins,prepare_plugins,1,
                    [{file,"src/rabbit_plugins.erl"},{line,202}]},
    {rabbit,broker_start,0,[{file,"src/rabbit.erl"},{line,293}]},
    {rabbit,start_it,1,[{file,"src/rabbit.erl"},{line,417}]},
    {init,start_em,1,[]},
    {init,do_boot,3,[]}]

{"init terminating in do_boot",{plugin_built_with_incompatible_erlang,"prometheus_process_collector"}}
init terminating in do_boot ()

Crash dump is being written to: erl_crash.dump...%                                                                                                                                                                               

I've updated RabbitMQ to 3.7.2 and downloaded the latest version of this rabbitmq_exporter, but when calling the metrics endpoint, I get a code 500 error. In the logging I see the following crash report:

2018-01-16 09:58:42.505 [error] <0.30093.30> CRASH REPORT Process <0.30093.30> with 0 neighbours crashed with reason: call to undefined function prometheus_cowboy2_handler:init(#{bindings => #{},body_length => 0,has_body => false,headers => #{...)
2018-01-16 09:58:42.506 [error] <0.30092.30> Ranch listener rabbit_web_dispatch_sup_15672, connection process <0.30092.30>, stream 1 had its request process <0.30093.30> exit with reason undef and stacktrace [{prometheus_cowboy2_handler,init,[#{bindings => #{},body_length => 0,has_body => false,headers => #{...},host => <<"...IP...">>,host_info => undefined,method => <<"GET">>,path => <<"/api/metrics">>,path_info => undefined,peer => {...IP...},pid => <0.30092.30>,port => 15672,qs => <<>>,ref => rabbit_web_dispatch_sup_15672,scheme => <<"http">>,streamid => 1,version => 'HTTP/1.0'},[]],[]},{cowboy_handler,execute,2,[{file,"src/cowboy_handler.erl"},{line,37}]},{cowboy_stream_h,execute,3,[{file,"src/cowboy_stream_h.erl"},{line,205}]},{cowboy_stream_h,request_process,3,[{file,"src/cowboy_stream_h.erl"},{line,184}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]

Any idea what this means? I've placed all the plugin files in the correct directory I think. At least it start correctly.

[xxx@xx plugins]# pwd
/usr/lib/rabbitmq/plugins
[xxx@xx plugins]# ls -alh
total 492K
drwxr-xr-x 2 root root 4.0K Jan 16 09:38 .
drwxr-xr-x 6 root root 4.0K Jan 14 15:45 ..
-rw-r--r-- 1 root root  14K Nov 21 22:37 accept-0.3.3.ez
-rw-r--r-- 1 root root 195K Dec 21 23:28 prometheus-3.4.5.ez
-rw-r--r-- 1 root root  15K Dec  9 13:07 prometheus_cowboy-0.1.4.ez
-rw-r--r-- 1 root root  21K Nov 21 22:37 prometheus_httpd-2.1.8.ez
-rw-r--r-- 1 root root  18K Dec 21 23:28 prometheus_process_collector-1.3.1.ez
-rw-r--r-- 1 root root 209K Dec 21 23:28 prometheus_rabbitmq_exporter-v3.7.1.1.ez

I've also upgraded the plugin by disabling and enabling it.

Hey Guys,

I am very very new to this and I have questions. Where would I ask such questions?

Thanks,
Michael

I get metrics at [ip address]:15672/api/metrics, but the only plugin that seemed to do anything is prometheus_rabbitmq_exporter. Is this expected behavior?

Here is the results of following the directions to the best of my understanding.

I have downloaded all the plugins for RabbitMQ 3.7.5 and put them in the plugin directory.

This is the contents of my plugin dir

ls -la /usr/lib/rabbitmq/lib/rabbitmq_server-3.7.5/plugins/
total 6132
drwxr-xr-x 2 root root 4096 Jun 6 20:52 .
drwxr-xr-x 8 root root 4096 Jun 6 07:03 ..
-rw-r--r-- 1 root root 13395 Jan 20 10:10 accept-0.3.3.ez
-rw-r--r-- 1 root root 104625 May 10 03:07 amqp10_client-3.7.5.ez
-rw-r--r-- 1 root root 44243 May 10 03:07 amqp10_common-3.7.5.ez
-rw-r--r-- 1 root root 411194 May 10 03:07 amqp_client-3.7.5.ez
-rw-r--r-- 1 root root 257478 May 10 03:07 cowboy-2.2.2.ez
-rw-r--r-- 1 root root 439901 May 10 03:07 cowlib-2.1.0.ez
-rw-r--r-- 1 root root 69246 May 10 03:07 goldrush-0.1.9.ez
-rw-r--r-- 1 root root 101517 May 10 03:07 jsx-2.8.2.ez
-rw-r--r-- 1 root root 222957 May 10 03:07 lager-3.5.1.ez
-rw-r--r-- 1 root root 198993 Jan 20 10:10 prometheus-3.4.5.ez
-rw-r--r-- 1 root root 14344 Jan 20 10:10 prometheus_cowboy-0.1.4.ez
-rw-r--r-- 1 root root 22055 Jan 20 10:10 prometheus_httpd-2.1.8.ez
-rw-r--r-- 1 root root 17468 Jan 20 10:10 prometheus_process_collector-1.3.1.ez
-rw-r--r-- 1 root root 215171 Jan 20 10:10 prometheus_rabbitmq_exporter-v3.7.2.1.ez
-rw-r--r-- 1 root root 704850 May 10 03:07 rabbit_common-3.7.5.ez
-rw-r--r-- 1 root root 256088 May 10 03:07 rabbitmq_amqp1_0-3.7.5.ez
-rw-r--r-- 1 root root 31486 May 10 03:07 rabbitmq_auth_backend_cache-3.7.5.ez
-rw-r--r-- 1 root root 14411 May 10 03:07 rabbitmq_auth_backend_http-3.7.5.ez
-rw-r--r-- 1 root root 49128 May 10 03:07 rabbitmq_auth_backend_ldap-3.7.5.ez
-rw-r--r-- 1 root root 20115 May 10 03:07 rabbitmq_auth_mechanism_ssl-3.7.5.ez
-rw-r--r-- 1 root root 67896 May 10 03:07 rabbitmq_aws-3.7.5.ez
-rw-r--r-- 1 root root 19056 May 10 03:07 rabbitmq_consistent_hash_exchange-3.7.5.ez
-rw-r--r-- 1 root root 16027 May 10 03:07 rabbitmq_event_exchange-3.7.5.ez
-rw-r--r-- 1 root root 237527 May 10 03:07 rabbitmq_federation-3.7.5.ez
-rw-r--r-- 1 root root 14433 May 10 03:07 rabbitmq_federation_management-3.7.5.ez
-rw-r--r-- 1 root root 27034 May 10 03:07 rabbitmq_jms_topic_exchange-3.7.5.ez
-rw-r--r-- 1 root root 1035205 May 10 03:07 rabbitmq_management-3.7.5.ez
-rw-r--r-- 1 root root 178911 May 10 03:07 rabbitmq_management_agent-3.7.5.ez
-rw-r--r-- 1 root root 156788 May 10 03:07 rabbitmq_mqtt-3.7.5.ez
-rw-r--r-- 1 root root 25826 May 10 03:07 rabbitmq_peer_discovery_aws-3.7.5.ez
-rw-r--r-- 1 root root 54687 May 10 03:07 rabbitmq_peer_discovery_common-3.7.5.ez
-rw-r--r-- 1 root root 50389 May 10 03:07 rabbitmq_peer_discovery_consul-3.7.5.ez
-rw-r--r-- 1 root root 33525 May 10 03:07 rabbitmq_peer_discovery_etcd-3.7.5.ez
-rw-r--r-- 1 root root 14978 May 10 03:07 rabbitmq_peer_discovery_k8s-3.7.5.ez
-rw-r--r-- 1 root root 7688 May 10 03:07 rabbitmq_random_exchange-3.7.5.ez
-rw-r--r-- 1 root root 19126 May 10 03:07 rabbitmq_recent_history_exchange-3.7.5.ez
-rw-r--r-- 1 root root 49772 May 10 03:07 rabbitmq_sharding-3.7.5.ez
-rw-r--r-- 1 root root 178100 May 10 03:07 rabbitmq_shovel-3.7.5.ez
-rw-r--r-- 1 root root 23970 May 10 03:07 rabbitmq_shovel_management-3.7.5.ez
-rw-r--r-- 1 root root 146120 May 10 03:07 rabbitmq_stomp-3.7.5.ez
-rw-r--r-- 1 root root 71968 May 10 03:07 rabbitmq_top-3.7.5.ez
-rw-r--r-- 1 root root 61771 May 10 03:07 rabbitmq_tracing-3.7.5.ez
-rw-r--r-- 1 root root 71542 May 10 03:07 rabbitmq_trust_store-3.7.5.ez
-rw-r--r-- 1 root root 44552 May 10 03:07 rabbitmq_web_dispatch-3.7.5.ez
-rw-r--r-- 1 root root 39263 May 10 03:07 rabbitmq_web_mqtt-3.7.5.ez
-rw-r--r-- 1 root root 66306 May 10 03:07 rabbitmq_web_mqtt_examples-3.7.5.ez
-rw-r--r-- 1 root root 52751 May 10 03:07 rabbitmq_web_stomp-3.7.5.ez
-rw-r--r-- 1 root root 51995 May 10 03:07 rabbitmq_web_stomp_examples-3.7.5.ez
-rw-r--r-- 1 root root 58315 May 10 03:07 ranch-1.5.0.ez
-rw-r--r-- 1 root root 36685 May 10 03:07 ranch_proxy_protocol-1.5.0.ez
-rw-r--r-- 1 root root 59 May 10 03:07 README
-rw-r--r-- 1 root root 55901 May 10 03:07 recon-2.3.2.ez

This is what rabbitmq-plugins reports

rabbitmq-plugins list
Configured: E = explicitly enabled; e = implicitly enabled
| Status: * = running on rabbit@[redacted]
|/
[ ] prometheus_rabbitmq_exporter v3.7.2.1
[ ] rabbitmq_amqp1_0 3.7.5
[ ] rabbitmq_auth_backend_cache 3.7.5
[ ] rabbitmq_auth_backend_http 3.7.5
[ ] rabbitmq_auth_backend_ldap 3.7.5
[ ] rabbitmq_auth_mechanism_ssl 3.7.5
[ ] rabbitmq_consistent_hash_exchange 3.7.5
[ ] rabbitmq_event_exchange 3.7.5
[ ] rabbitmq_federation 3.7.5
[ ] rabbitmq_federation_management 3.7.5
[ ] rabbitmq_jms_topic_exchange 3.7.5
[E*] rabbitmq_management 3.7.5
[e*] rabbitmq_management_agent 3.7.5
[ ] rabbitmq_mqtt 3.7.5
[ ] rabbitmq_peer_discovery_aws 3.7.5
[e*] rabbitmq_peer_discovery_common 3.7.5
[E*] rabbitmq_peer_discovery_consul 3.7.5
[ ] rabbitmq_peer_discovery_etcd 3.7.5
[ ] rabbitmq_peer_discovery_k8s 3.7.5
[ ] rabbitmq_random_exchange 3.7.5
[ ] rabbitmq_recent_history_exchange 3.7.5
[ ] rabbitmq_sharding 3.7.5
[ ] rabbitmq_shovel 3.7.5
[ ] rabbitmq_shovel_management 3.7.5
[ ] rabbitmq_stomp 3.7.5
[ ] rabbitmq_top 3.7.5
[ ] rabbitmq_tracing 3.7.5
[ ] rabbitmq_trust_store 3.7.5
[e*] rabbitmq_web_dispatch 3.7.5
[ ] rabbitmq_web_mqtt 3.7.5
[ ] rabbitmq_web_mqtt_examples 3.7.5
[ ] rabbitmq_web_stomp 3.7.5
[ ] rabbitmq_web_stomp_examples 3.7.5`

None of these plugins seem to do anything when I try to enable them

rabbitmq-plugins enable accept
The following plugins have been configured:
rabbitmq_management
rabbitmq_management_agent
rabbitmq_peer_discovery_common
rabbitmq_peer_discovery_consul
rabbitmq_web_dispatch
Applying plugin configuration to rabbit@[redacted]
Plugin configuration unchanged.

rabbitmq-plugins enable prometheus
The following plugins have been configured:
rabbitmq_management
rabbitmq_management_agent
rabbitmq_peer_discovery_common
rabbitmq_peer_discovery_consul
rabbitmq_web_dispatch
Applying plugin configuration to rabbit@[redacted]
Plugin configuration unchanged.

rabbitmq-plugins enable prometheus_cowboy
The following plugins have been configured:
rabbitmq_management
rabbitmq_management_agent
rabbitmq_peer_discovery_common
rabbitmq_peer_discovery_consul
rabbitmq_web_dispatch
Applying plugin configuration to rabbit@[redacted]
Plugin configuration unchanged.

rabbitmq-plugins enable prometheus_httpd
The following plugins have been configured:
rabbitmq_management
rabbitmq_management_agent
rabbitmq_peer_discovery_common
rabbitmq_peer_discovery_consul
rabbitmq_web_dispatch
Applying plugin configuration to rabbit@[redacted]
Plugin configuration unchanged.

This works:

rabbitmq-plugins enable prometheus_rabbitmq_exporter
The following plugins have been configured:
prometheus_rabbitmq_exporter
rabbitmq_management
rabbitmq_management_agent
rabbitmq_peer_discovery_common
rabbitmq_peer_discovery_consul
rabbitmq_web_dispatch
Applying plugin configuration to rabbit@[redacted]
The following plugins have been enabled:
prometheus_rabbitmq_exporter

and this doesn't

rabbitmq-plugins enable prometheus_process_collector
The following plugins have been configured:
prometheus_rabbitmq_exporter
rabbitmq_management
rabbitmq_management_agent
rabbitmq_peer_discovery_common
rabbitmq_peer_discovery_consul
rabbitmq_web_dispatch
Applying plugin configuration to rabbit@[redacted]
Plugin configuration unchanged.

I upgraded a rabbitmq cluster (3 nodes) from version 3.3.5 to version 3.6.5 in order to use the prometheus exporter. After the upgrade I managed to enable the prometheus_rabbitmq_exporter and prometheus plugins but the prometheus_process_collector plugin fails to load with the following error.

=WARNING REPORT==== 24-Aug-2016::13:24:15 ===
The on_load function for module prometheus_process_collector returned {error,
{bad_lib,
"Library version (2.11) not compatible (with 2.4)."}}

With the last version I get errors by prometheus:
text format parsing error in line 1589: expected float as value, got "''"

The offending line is
rabbitmq_queue_consumer_utilisation{vhost="hermesmo",queue="rejectedMessagesQueue"} ''
The queues haven't got any traffic yet.

There is only one plugin vissible in list after installing plugins to Rabbbi v3.7.7.
[E*] prometheus_rabbitmq_exporter v3.7.2.2

Used plugins: https://github.com/deadtrickster/prometheus_rabbitmq_exporter/releases/tag/v3.7.2.2
(Ubuntu 16.04.4 LTS)

Could you provide compatible plugin?

In the README there is a config example that states

%% Be careful to declare the "prometheus" app
%% environment before the "rabbitmq_management" one.

How exactly can configuration "section" order matter? The entire config file is loaded and parsed at once. If this plugin depends on something in rabbitmq_management then it must declare a dependency on it.

On a somewhat related note I think it's time to produce a 3.7.x config schema and examples. 3.6.x has been EOL'd in May.

We have a rabbitmq 3.6.6 with these plugins:

[e*] amqp_client                       3.6.6
[  ] cowboy                            1.0.3
[  ] cowlib                            1.0.1
[e*] mochiweb                          2.13.1
[E*] prometheus                        3.0.1
[E*] prometheus_process_collector      1.0.0
[E*] prometheus_rabbitmq_exporter      v3.6.5.6
[  ] rabbitmq_amqp1_0                  3.6.6
[  ] rabbitmq_auth_backend_ldap        3.6.6
[  ] rabbitmq_auth_mechanism_ssl       3.6.6
[  ] rabbitmq_consistent_hash_exchange 3.6.6
[  ] rabbitmq_event_exchange           3.6.6
[  ] rabbitmq_federation               3.6.6
[  ] rabbitmq_federation_management    3.6.6
[  ] rabbitmq_jms_topic_exchange       3.6.6
[E*] rabbitmq_management               3.6.6
[e*] rabbitmq_management_agent         3.6.6
[  ] rabbitmq_management_visualiser    3.6.6
[  ] rabbitmq_mqtt                     3.6.6
[  ] rabbitmq_recent_history_exchange  1.2.1
[  ] rabbitmq_sharding                 0.1.0
[  ] rabbitmq_shovel                   3.6.6
[  ] rabbitmq_shovel_management        3.6.6
[  ] rabbitmq_stomp                    3.6.6
[  ] rabbitmq_top                      3.6.6
[  ] rabbitmq_tracing                  3.6.6
[  ] rabbitmq_trust_store              3.6.6
[e*] rabbitmq_web_dispatch             3.6.6
[  ] rabbitmq_web_stomp                3.6.6
[  ] rabbitmq_web_stomp_examples       3.6.6
[  ] sockjs                            0.3.4
[e*] webmachine                        1.10.3

A colleague experimented with the rabbitmq_sharding plugin: He enable it, disabled it and restarted the server. The server then refused to start with this error:

Error description:
   {could_not_start,rabbitmq_management,
       {badarg,
           [{ets,insert,
                [prometheus_registry_table,{default,prometheus_summary}],
                []},
            {prometheus_registry,register_collector,2,
                [{file,"src/prometheus_registry.erl"},{line,79}]},
            {prometheus_metric,insert_mf,3,
                [{file,"src/prometheus_metric.erl"},{line,101}]},
            {prometheus_rabbitmq_exporter,dispatcher,0,
                [{file,"src/prometheus_rabbitmq_exporter.erl"},{line,12}]},
            {rabbit_mgmt_dispatcher,'-build_dispatcher/1-lc$^0/1-1-',1,
                [{file,"src/rabbit_mgmt_dispatcher.erl"},{line,27}]},
            {rabbit_mgmt_dispatcher,'-build_dispatcher/1-lc$^0/1-1-',1,
                [{file,"src/rabbit_mgmt_dispatcher.erl"},{line,27}]},
            {rabbit_mgmt_dispatcher,build_dispatcher,1,
                [{file,"src/rabbit_mgmt_dispatcher.erl"},{line,27}]},
            {rabbit_mgmt_app,make_loop,1,
                [{file,"src/rabbit_mgmt_app.erl"},{line,57}]}]}}

We somehow "fixed" this by disabling the rabbitmq_management plugin (which also disabled the prometheus_rabbitmq_exporter plugin). Then the server at least started again, but it refused to start with prometheus_rabbitmq_exporter.
We then successfully enabled first the sharding then the prometheus_rabbitmq_exporter; the server survived a restart.
We also could disable rabbitmq_sharding and restart, so we are now at the step before the experiments with sharding.

Do you have idea what confused the exporter plugin?

Status of node 'rabbit@stage-icms-rabbitmq1' ...
[{pid,31654},
 {running_applications,
     [{prometheus_rabbitmq_exporter,
          "RabbitMQ Prometheus.io metrics exporter","v3.6.5.6"},
      {rabbitmq_management,"RabbitMQ Management Console","3.6.6"},
      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.6.6"},
      {webmachine,"webmachine","1.10.3"},
      {mochiweb,"MochiMedia Web Server","2.13.1"},
      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.6.6"},
      {rabbit,"RabbitMQ","3.6.6"},
      {os_mon,"CPO  CXC 138 46","2.4.1"},
      {amqp_client,"RabbitMQ AMQP Client","3.6.6"},
      {prometheus_process_collector,
          "Prometheus.io process collector\n    Collector exports the current state of process metrics including cpu, memory,\n    file descriptor usage and native threads count as well as the process start and up times.",
          "1.0.0"},
      {prometheus,"Prometheus.io client in Erlang","3.0.1"},
      {mnesia,"MNESIA  CXC 138 12","4.14.2"},
      {rabbit_common,[],"3.6.6"},
      {ssl,"Erlang/OTP SSL application","8.1"},
      {public_key,"Public key infrastructure","1.3"},
      {ranch,"Socket acceptor pool for TCP protocols.","1.2.1"},
      {inets,"INETS  CXC 138 49","6.3.4"},
      {crypto,"CRYPTO","3.7.2"},
      {syntax_tools,"Syntax tools","2.1.1"},
      {asn1,"The Erlang ASN1 compiler version 4.0.4","4.0.4"},
      {compiler,"ERTS  CXC 138 10","7.0.3"},
      {xmerl,"XML parser","1.3.12"},
      {sasl,"SASL  CXC 138 11","3.0.2"},
      {stdlib,"ERTS  CXC 138 10","3.2"},
      {kernel,"ERTS  CXC 138 10","5.1.1"}]},
 {os,{unix,linux}},
 {erlang_version,
     "Erlang/OTP 19 [erts-8.2] [source] [64-bit] [smp:16:16] [async-threads:256] [kernel-poll:true]\n"},
 {memory,
     [{total,89240368},
      {connection_readers,459288},
      {connection_writers,238032},
      {connection_channels,969744},
      {connection_other,3020744},
      {queue_procs,1175072},
      {queue_slave_procs,0},
      {plugins,3405216},
      {other_proc,16810208},
      {mnesia,375952},
      {mgmt_db,5084944},
      {msg_index,77456},
      {other_ets,1871768},
      {binary,11048176},
      {code,25202610},
      {atom,1041593},
      {other_system,18459565}]},
 {alarms,[]},
 {listeners,[{clustering,25672,"::"},{amqp,5672,"::"}]},
 {vm_memory_high_watermark,0.8},
 {vm_memory_limit,6714405683},
 {disk_free_limit,50000000},
 {disk_free,48178319360},
 {file_descriptors,
     [{total_limit,924},
      {total_used,15},
      {sockets_limit,829},
      {sockets_used,13}]},
 {processes,[{limit,1048576},{used,1216}]},
 {run_queue,0},
 {uptime,814},
 {kernel,{net_ticktime,60}}]

Hi,

Thanks for your work on this project !

Could you please add some metrics about the cluster state ? Especially about partitions ?

I am making an alternative version of your sample grafana dashboard using the metrics exported by this plugin (and related such as prometheus_process_collector), dashboard that I will be happy to share back in a PR.

A problem I have is that the rabbitmq instances are identified by their "instance" label, which is an ephemeral ipaddress and port.

Since I deploy rabbitmq as docker containers inside kubernetes, I would like another label to be added which is more stable than "instance", for example the hostname (of the container) or even better the rabbitmq nodename.

In other words, when I query process_uptime_seconds in prometheus, I get:

process_uptime_seconds{instance="10.129.0.18:15672",job="kubernetes-service-endpoints",kubernetes_name="rabbitmq",kubernetes_namespace="backends"}	3656

I would like to have:

process_uptime_seconds{instance="10.129.0.18:15672",nodename="[email protected]",job="kubernetes-service-endpoints",kubernetes_name="rabbitmq",kubernetes_namespace="backends"}	3656

The problem with the "instance" label is that I cannot differentiate:

• a single rabbitmq instance being redeployed and getting another IP address over time (then I can safely workaround by wrapping the query in sum (expression) without (instance)
• a multi instance rabbitmq cluster where I would like to track the memory usage, file descriptors etc. by nodename

If I don't wrap the data in sum (expression) without (instance) then I get "multiple value error" when displaying single stat such as uptime in the dashboard, and I get the same label multiple times in the state displaying the different kind of messages (acknowledged, redelivered etc).

I would also like to templatize the dashboard so that I can check the stats of each rabbitmq node separately in the case of a multi-cluster node.

Hello there,

I'm trying to disable metrics collected for queues and exchanges (e.g. rabbitmq_queue_disk_reads).

Ideally I'm able to disable only for certain queues, however it would be alright to disable any queue related metrics.

(The reason is we have very high number of queues and the metrics take away significant performance from our brokers.)

So far I've tried:

[...]
{ prometheus, [
  { rabbitmq_exporter, [
    { path, "metrics" },
    { queue_messages_stat, [ ] },
    { exchange_messages_stat, [] }
  ] }
] },
[...]

Is there a simple way to disable any queue related metrics?

Thanks in advance for your help :)

Cheers,
Thubo

On a 3-node clustered rabbitmq version 3.6.11, I enabled prometheus plugins on all three nodes
I see all the beam metrics, but not any of the rabbitmq metrics

Initially tried
prometheus_rabbitmq_exporter v3.6.12.1

then tried
prometheus_rabbitmq_exporter v3.6.9.1

there are no errors in rabbit logs.
prometheus metrics are working.. just missing the rabbitmq metrics

thank you

I'm deploying a Rabbit cluster and the plugin works fine on the first node that its deployed on but all subsequent nodes return:

{"error":"Internal Server Error","reason":"{error,\n {error,badarg,\n [{ets,select,\n [connection_stats,\n [{{{'','$1'},'$2',''},[{'==',create,'$1'}],['$2']}]],\n []},\n {prometheus_rabbitmq_overview_collector,created_events,1,\n [{file,\n "src/collectors/prometheus_rabbitmq_overview_collector.erl"},\n {line,63}]},\n {prometheus_rabbitmq_overview_collector,collect_metrics,2,\n [{file,\n "src/collectors/prometheus_rabbitmq_overview_collector.erl"},\n {line,39}]},\n {prometheus_model_helpers,create_mf,5,\n [{file,"src/model/prometheus_model_helpers.erl"},{line,28}]},\n {prometheus_rabbitmq_overview_collector,collect_mf,2,\n [{file,\n "src/collectors/prometheus_rabbitmq_overview_collector.erl"},\n {line,32}]},\n {prometheus_registry,'-collect/2-lc$^0/1-0-',3,\n [{file,"src/prometheus_registry.erl"},{line,14}]},\n {prometheus_registry,'-collect/2-lc$^0/1-0-',3,\n [{file,"src/prometheus_registry.erl"},{line,14}]},\n {prometheus_text_format,format,1,\n [{file,"src/formats/prometheus_text_format.erl"},{line,28}]}]}}\n"}

Am I doing something wrong? (All the cluster nodes are configured identically through an ansible playbook).

Even though the plugin is enabled, the node itself is not listening for requests on 9419 or collecting.

root@ip-10-2-249-45:~#
curl localhost:9419/metrics
curl: (7) Failed to connect to localhost port 9419: Connection refused

root@ip-10-2-249-45:~# rabbitmq-plugins enable prometheus_rabbitmq_exporter
The following plugins have been configured:
prometheus_rabbitmq_exporter
rabbitmq_management
rabbitmq_management_agent
rabbitmq_peer_discovery_aws
rabbitmq_peer_discovery_common
rabbitmq_web_dispatch
Applying plugin configuration to [email protected]...
Plugin configuration unchanged.

root@ip-10-2-249-45:~#
[E*] prometheus_rabbitmq_exporter 3.7.2.3
[ ] rabbitmq_amqp1_0 3.7.8 [0/906]
[ ] rabbitmq_auth_backend_cache 3.7.8
[ ] rabbitmq_auth_backend_http 3.7.8
[ ] rabbitmq_auth_backend_ldap 3.7.8
[ ] rabbitmq_auth_mechanism_ssl 3.7.8
[ ] rabbitmq_consistent_hash_exchange 3.7.8
[ ] rabbitmq_event_exchange 3.7.8
[ ] rabbitmq_federation 3.7.8
[ ] rabbitmq_federation_management 3.7.8
[ ] rabbitmq_jms_topic_exchange 3.7.8
[E*] rabbitmq_management 3.7.8
[e*] rabbitmq_management_agent 3.7.8
[ ] rabbitmq_mqtt 3.7.8
[E*] rabbitmq_peer_discovery_aws 3.7.8
[e*] rabbitmq_peer_discovery_common 3.7.8
[ ] rabbitmq_peer_discovery_consul 3.7.8
[ ] rabbitmq_peer_discovery_etcd 3.7.8
[ ] rabbitmq_peer_discovery_k8s 3.7.8
[ ] rabbitmq_random_exchange 3.7.8
[ ] rabbitmq_recent_history_exchange 3.7.8
[ ] rabbitmq_sharding 3.7.8
[ ] rabbitmq_shovel 3.7.8
[ ] rabbitmq_shovel_management 3.7.8
[ ] rabbitmq_stomp 3.7.8
[ ] rabbitmq_top 3.7.8
[ ] rabbitmq_tracing 3.7.8
[ ] rabbitmq_trust_store 3.7.8
[e*] rabbitmq_web_dispatch 3.7.8
[ ] rabbitmq_web_mqtt 3.7.8
[ ] rabbitmq_web_mqtt_examples 3.7.8
[ ] rabbitmq_web_stomp 3.7.8
[ ] rabbitmq_web_stomp_examples 3.7.8`

root@ip-10-2-249-45:~#
netstat -tulnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:25672 0.0.0.0:* LISTEN 2685/beam.smp
tcp 0 0 0.0.0.0:4369 0.0.0.0:* LISTEN 2787/epmd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1171/sshd
tcp 0 0 0.0.0.0:15672 0.0.0.0:* LISTEN 2685/beam.smp
tcp6 0 0 :::5672 :::* LISTEN 2685/beam.smp
tcp6 0 0 :::9100 :::* LISTEN 1145/node_exporter
tcp6 0 0 :::4369 :::* LISTEN 2787/epmd
tcp6 0 0 :::22 :::* LISTEN 1171/sshd
udp 0 0 0.0.0.0:68 0.0.0.0:* 945/dhclient
udp 0 0 0.0.0.0:46175 0.0.0.0:* 2685/beam.smp ]

[ root@ip-10-2-249-45:~#
systemctl status rabbitmq-server.service
rabbitmq-server.service - RabbitMQ broker
Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2018-10-22 10:03:41 EDT; 7min ago
Process: 2573 ExecStop=/usr/lib/rabbitmq/bin/rabbitmqctl shutdown (code=exited, status=0/SUCCESS)
Main PID: 2685 (beam.smp)
Status: "Initialized"
Tasks: 85
Memory: 81.8M
CPU: 10.667s
CGroup: /system.slice/rabbitmq-server.service
├─2685 /usr/lib/erlang/erts-10.1.1/bin/beam.smp -W w -A 64 -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -K true -- -root /usr/l
├─2787 /usr/lib/erlang/erts-10.1.1/bin/epmd -daemon
├─2929 erl_child_setup 32768
├─2970 inet_gethost 4
└─2971 inet_gethost 4

Oct 22 10:03:41 ip-10-2-249-45 rabbitmq-server[2685]: ## ##
Oct 22 10:03:41 ip-10-2-249-45 rabbitmq-server[2685]: ## ## RabbitMQ 3.7.8. Copyright (C) 2007-2018 Pivotal Software, Inc.
Oct 22 10:03:41 ip-10-2-249-45 rabbitmq-server[2685]: ########## Licensed under the MPL. See http://www.rabbitmq.com/
Oct 22 10:03:41 ip-10-2-249-45 rabbitmq-server[2685]: ###### ##
Oct 22 10:03:41 ip-10-2-249-45 rabbitmq-server[2685]: ########## Logs: /var/log/rabbitmq/[email protected]
Oct 22 10:03:41 ip-10-2-249-45 rabbitmq-server[2685]: /var/log/rabbitmq/[email protected]_upgrade.log
Oct 22 10:03:41 ip-10-2-249-45 rabbitmq-server[2685]: Starting broker...
Oct 22 10:03:41 ip-10-2-249-45 rabbitmq-server[2685]: systemd unit for activation check: "rabbitmq-server.service"
Oct 22 10:03:41 ip-10-2-249-45 systemd[1]: Started RabbitMQ broker.
Oct 22 10:03:41 ip-10-2-249-45 rabbitmq-server[2685]: completed with 6 plugins. ]

The documentation has a section on configuring the plugin https://github.com/deadtrickster/prometheus_rabbitmq_exporter#configuration

My goal is to configure the plugin to use the rabbitmq node name as the "instance" label, as you hinted in #23 (comment)

From what I understand, we have to configure the rabbitmq_exporter options through the environment of the prometheus application. To configure a custom metrics path I expect to be able to do this:

[
        { rabbit, [
                { loopback_users, [ ] },
                { tcp_listeners, [ 5672 ] },
                { ssl_listeners, [ ] },
                { hipe_compile, false }
        ] },
        { rabbitmq_management, [ { listener, [
                { port, 15672 },
                { ssl, false }
        ] } ] },
        { prometheus, [ { rabbitmq_exporter, [
                { path, "/mymetrics" }
        ] } ] }
].

If I save the above config as "rabbitmq.config" then run:

 docker run --rm -ti -p 15672:15672 -v $PWD/rabbitmq.config:/etc/rabbitmq/rabbitmq.config:ro deadtrickster/rabbitmq_prometheus:3.6.9.1

Then rabbitmq crashes with the following message:

=CRASH REPORT==== 13-Apr-2017::08:19:05 ===
  crasher:
    initial call: application_master:init/4
    pid: <0.438.0>
    registered_name: []
    exception exit: {bad_return,
                     {{rabbit_mgmt_app,start,[normal,[]]},
                      {'EXIT',
                       {badarg,
                        [{ets,insert,
                          [prometheus_registry_table,
                           {default,prometheus_summary}],
                          []},
                         {prometheus_registry,register_collector,2,
                          [{file,"src/prometheus_registry.erl"},{line,99}]},
                         {prometheus_metric,insert_mf,3,
                          [{file,"src/prometheus_metric.erl"},{line,101}]},
                         {prometheus_http,setup,0,
                          [{file,"src/prometheus_http.erl"},{line,53}]},
                         {prometheus_rabbitmq_exporter,dispatcher,0,
                          [{file,"src/prometheus_rabbitmq_exporter.erl"},
                           {line,11}]},
                         {rabbit_mgmt_dispatcher,
                          '-build_dispatcher/1-lc$^2/1-2-',1,
                          [{file,"src/rabbit_mgmt_dispatcher.erl"},{line,32}]},
                         {rabbit_mgmt_dispatcher,
                          '-build_dispatcher/1-lc$^2/1-2-',1,
                          [{file,"src/rabbit_mgmt_dispatcher.erl"},{line,33}]},
                         {rabbit_mgmt_dispatcher,build_dispatcher,1,
                          [{file,"src/rabbit_mgmt_dispatcher.erl"},
                           {line,32}]}]}}}}
      in function  application_master:init/4 (application_master.erl, line 134)
    ancestors: [<0.437.0>]
    messages: [{'EXIT',<0.439.0>,normal}]
    links: [<0.437.0>,<0.31.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 1598
    stack_size: 27
    reductions: 170
  neighbours:

What did I do wrong?

If prometheus_rabbitmq_exporter is enabled before the node is restarted, the rabbitmq_management API stops returning queue metrics:

I am not quiet sure how to get to the bottom of this, but I have an environment where I can reproduce this relatively easy. I have observed the same behaviour semi-consistently in the following configurations:

• Ubuntu 14.04.5 LTS
• RabbitMQ 3.6.15 with Erlang/OTP 19.3.x or 20.3.x
• RabbitMQ 3.7.5 with Erlang/OTP 19.3.x or 20.3.x
• prometheus_rabbitmq_exporter 3.6.12.1, 3.6.14.1 or 3.7.2.1
• prometheus.erl 3.4.0, 3.4.5 or 3.5.0

There are no crashes of any sort, all processes are healthy and the supervision tree looks correct. Any ideas on how I might narrow down this issue?

cc @michaelklishin @essen

I'm interested to get some stats about what's visible here:

• File descriptors: value process_open_fds / max process_max_fds
• Socket descriptors: ???
• Erlang processes: value erlang_vm_process_count / max erlang_vm_process_limit
• Memory: value sum by (instance) (erlang_vm_memory_bytes_total) / max ???
• Disp Space: value ??rabbitmq_queue_disk_size_bytes?? / max ???

Have I missed something or is it not possible to get all of these data?

Also in your description you mention:

Process Metrics
(Process info collector must be enabled)

Could you describe how to enable the process info collector?

Finally, I think there's a typo in the image: https://camo.githubusercontent.com/c33c1c497eba9a4125b2c84fb1c89ee9972be12c/687474703a2f2f692e696d6775722e636f6d2f745769447735362e706e673f31
Bottom left "Virtual Memory"

According to the README, this is how the plugin should be enabled:

 rabbitmq-plugins enable accept
 rabbitmq-plugins enable prometheus
 rabbitmq-plugins enable prometheus_cowboy
 rabbitmq-plugins enable prometheus_httpd
 rabbitmq-plugins enable prometheus_rabbitmq_exporter

As of RabbitMQ 3.7.0 — which has been around for some 9 months now — dependencies such as prometheus_cowboy will not show up in the rabbitmq-plugins list unless they explicitly depend on the rabbit app.

Assuming correct dependency declaration in those projects, is it really necessary to ask the user to enable them explicitly, especially if only prometheus_rabbitmq_exporter would show up on the list?

I'm using rabbitmq-3.6.6 (from dockerhub) with the following plugins:

autocluster-0.6.1.ez (from master)
rabbitmq_aws-0.1.3.ez (from master)
prometheus-3.0.1.ez
prometheus_process_collector-1.0.0.ez
prometheus_rabbitmq_exporter-v3.6.5.6.ez

As soon as I hit /api/metrics rabbitmq segfaults. Platform is kubernetes. I'm not sure which faults it is, just opening the bug here in case anyone runs into the same problem

           PID: 24250 (beam.smp)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Thu 2016-12-01 14:09:42 UTC (2min 49s ago)
  Command Line: /usr/lib/erlang/erts-8.0.5/bin/beam.smp -W w -A 64 -P 1048576 -t 5000000 -stbt db -zdbbl 32000 -K true -B i -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.6.6/ebin -noshell -noinput -s rabbit boot -name [email protected] -boot start_sasl -config /etc/rabbitmq-sxris/rabbitmq -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger tty -rabbit error_logger tty -rabbit sasl_error_logger tty -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.6/plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/[email protected]" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/[email protected]" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672
    Executable: /usr/lib/erlang/erts-8.0.5/bin/beam.smp
 Control Group: /
         Slice: -.slice
       Boot ID: 3b17e7f05a894cc0b77331744b33a795
    Machine ID: 1a8d01d73f6744fc858f25a25be315f8
      Hostname: ip-10-43-128-107.eu-central-1.compute.internal
      Coredump: /var/lib/systemd/coredump/core.beam\x2esmp.0.3b17e7f05a894cc0b77331744b33a795.24250.1480601382000000000000.xz
       Message: Process 24250 (beam.smp) of user 0 dumped core.

How can I help solving the issue?

We're using rabbit 3.6.5 with your 3.6.5.9 release and something seems to not be working correctly.

I'm trying to create a grafana dashboard showing publish/delivery rates and all my queue's are always showing 0 (using the standard rabbit management interface I can see there is traffic going through my queues) but the prometheus counters are always showing 0.

I added a test queue and published a bunch of messages to the queue. I see the value show in the gauge for ready messages (and other metrics for that queue showing data)
ie:
rabbitmq_queue_messages_ready{vhost="/",queue="test"} 32
rabbitmq_queue_messages{vhost="/",queue="test"} 32
rabbitmq_queue_disk_writes{vhost="/",queue="test"} 32

But all the counters that should allow me to calculate rates are always returning zeros:
rabbitmq_queue_messages_published_total{vhost="/",queue="test"} 0
rabbitmq_queue_messages_delivered_total{vhost="/",queue="test"} 0
....
rabbitmq_queue_messages_returned_total{vhost="/",queue="test"} 0

I have no custom config in rabbitmq.conf for the prometheus_rabbitmq_exporter our enabled plugins are:
[autocluster,prometheus,prometheus_rabbitmq_exporter,rabbitmq_management]

(we're using AWS tag based autoclustering).

Thanks

Hi,
after I've deployed the plugins, what should I use as a target for prometheus? server:8080 gives a connection refused and on server:15672/metrics the server returns a 404.

Thanks in advance