prometheus_rabbitmq_exporter prevents queue metrics from working correctly via rabbitmq_management when the node restarts about prometheus_rabbitmq_exporter HOT 8 CLOSED

Btw, I have confirmed that queue_metrics_metrics_collector is running via erlang:process_info(erlang:whereis(queue_metrics_metrics_collector)) when this issue happens, as well as double-checked that rabbit_queue Mnesia table is tracking the correct pids for queue processes.

from prometheus_rabbitmq_exporter.

One interesting fact worth mentioning is that the problem goes away if the node gets restarted once or twice. This makes me believe that there is some sort of a race condition, maybe prometheus_rabbitmq_exporter is preventing a rabbitmq process registering correctly?

from prometheus_rabbitmq_exporter.

If it's a race then my first guess would be that prometheus scrapes before backend initialized properly by rabbit itself but you say everything is alive.

So how I can reproduce this exactly? I'm sure I've restarted node with this plugin enabled but I never noticed what you describing here. Maybe I wasn't looking at the right place...

from prometheus_rabbitmq_exporter.

@gerhard maybe we can arrange access for @deadtrickster to a one-off BOSH-managed deployment that reproduces the issue?

from prometheus_rabbitmq_exporter.

I've tried to reproduce this but couldn't yet, still trying.

I'm running 5 instances of perf-test 2.1.0 using the following flags: --uris amqp://USER:PASS@IP:5672/%2F --size 1000 --interval 60 --consumers 100 --producers 100 --auto-delete false --flag persistent --confirm 10 --confirm-timeout 30 --qos 1 --queue-pattern %d --queue-pattern-from 1 --queue-pattern-to 20 --queue-args x-max-length=1000 --rate 2 --consumer-rate 1

Attaching rabbitmqctl report: rabbitmqctl_report.log. In this case, I'm using 10 instances of perf-test, but I could reproduce the issue with just 5 instances of perf-test initially.

from prometheus_rabbitmq_exporter.

What should I see in rabbtmqctl_report.log? What I've spotted:

• app envs,
• conns,
• channels,
• exchanges,
• queues.

from prometheus_rabbitmq_exporter.

This is not a prometheus issue, but a race condition on RMQ boot steps. When collect_statistics is set to none and the management plugin is enabled, we force the collection at runtime by setting collect_statistics to fine in the management plugin boot steps. It seems that in some situations, this happens after the queues are started. We'll work on this issue.

from prometheus_rabbitmq_exporter.

Closing this since it's not related to prometheus_rabbitmq_exporter. Thank you!

from prometheus_rabbitmq_exporter.