Upgrade RabbitMQ on ocp13 and ocp23 like we do Elasticsearch about deploy HOT 3 CLOSED

To-do:

• Set up monitoring to alert when RabbitMQ updates are released
• Write patching process
• Put packages on hold

@jpmckinney, Would it help if we stopped the Docker application while RabbitMQ is patched? or would this create other alert?

from deploy.

Stopping Docker should be fine, as the application is designed to shutdown gracefully and survive restarts.

I spent a bit of time yesterday looking into an asynchronous RabbitMQ client, and it might not be so much work – I'll do some testing today, and see if it's promising. If so, we can skip this issue.

from deploy.

I successfully updated our RabbitMQ client to work asynchronously and survive RabbitMQ restarts, so this issue is resolved.

I just need to update the client on data-registry and kingfisher-collect (already did kingfisher-process, pelican-backend, pelican-frontend). I'm still using a blocking (synchronous) client for short-lived message publication, as otherwise I need to run the client's IO loop in a new thread, which is extra complexity for an unlikely scenario (i.e. RabbitMQ restarts during one of those short-lived connections).

The rest is just some notes to remind myself and others at a future date why the connection delay is 15s.

Looking at a recent RabbitMQ restart, I see (abbreviated):

08:58:25 RabbitMQ is asked to stop...
08:58:26 Stopping application 'rabbit'
08:58:26 stopped TCP listener on [::]:5672
08:58:26 [error] Error on AMQP connection <0.35485.0> (172.24.0.2:38242 -> 172.17.0.1:5672, vhost: '/', user: 'pelican_backend', state: running), channel 0:
08:58:26 [error] operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
... more of the same within 1 millisecond, then a 5 second gap in log messages ...
08:58:31 [notice] Application rabbit exited with reason: stopped
08:58:31 [info] Successfully stopped RabbitMQ and its dependencies
08:58:31 [info] Halting Erlang VM with the following applications:
08:58:35 [notice] Logging: configured log handlers are now ACTIVE
08:58:35 [info] Starting RabbitMQ 3.12.1 on Erlang 26.0.2 [jit]
08:58:36[ info] Ready to start client connection listeners
08:58:36[ info] started TCP listener on [::]:5672
08:58:36[ info] Server startup complete; 3 plugins started.

So, it's 10 seconds from the exception being received by the client, to new connections being possible. The client defaults to reconnecting every 15 seconds.

We can make it attempt to reconnect every 1 second, and therefore reconnect 4-5s faster.

However, we presently log connection failures at ERROR level, because if, e.g., RabbitMQ is down for an extended period, we want to know (while we have other ways of monitoring RabbitMQ, we don't necessarily have them in all cases).

So, to avoid having spurious error messages logged to Sentry, we can use a longer connection delay.

from deploy.