Comments (3)
To-do:
- Set up monitoring to alert when RabbitMQ updates are released
- Write patching process
- Put packages on hold
@jpmckinney, Would it help if we stopped the Docker application while RabbitMQ is patched? or would this create other alert?
from deploy.
Stopping Docker should be fine, as the application is designed to shutdown gracefully and survive restarts.
I spent a bit of time yesterday looking into an asynchronous RabbitMQ client, and it might not be so much work – I'll do some testing today, and see if it's promising. If so, we can skip this issue.
from deploy.
I successfully updated our RabbitMQ client to work asynchronously and survive RabbitMQ restarts, so this issue is resolved.
I just need to update the client on data-registry and kingfisher-collect (already did kingfisher-process, pelican-backend, pelican-frontend). I'm still using a blocking (synchronous) client for short-lived message publication, as otherwise I need to run the client's IO loop in a new thread, which is extra complexity for an unlikely scenario (i.e. RabbitMQ restarts during one of those short-lived connections).
The rest is just some notes to remind myself and others at a future date why the connection delay is 15s.
Looking at a recent RabbitMQ restart, I see (abbreviated):
08:58:25 RabbitMQ is asked to stop...
08:58:26 Stopping application 'rabbit'
08:58:26 stopped TCP listener on [::]:5672
08:58:26 [error] Error on AMQP connection <0.35485.0> (172.24.0.2:38242 -> 172.17.0.1:5672, vhost: '/', user: 'pelican_backend', state: running), channel 0:
08:58:26 [error] operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
... more of the same within 1 millisecond, then a 5 second gap in log messages ...
08:58:31 [notice] Application rabbit exited with reason: stopped
08:58:31 [info] Successfully stopped RabbitMQ and its dependencies
08:58:31 [info] Halting Erlang VM with the following applications:
08:58:35 [notice] Logging: configured log handlers are now ACTIVE
08:58:35 [info] Starting RabbitMQ 3.12.1 on Erlang 26.0.2 [jit]
08:58:36[ info] Ready to start client connection listeners
08:58:36[ info] started TCP listener on [::]:5672
08:58:36[ info] Server startup complete; 3 plugins started.
So, it's 10 seconds from the exception being received by the client, to new connections being possible. The client defaults to reconnecting every 15 seconds.
We can make it attempt to reconnect every 1 second, and therefore reconnect 4-5s faster.
However, we presently log connection failures at ERROR level, because if, e.g., RabbitMQ is down for an extended period, we want to know (while we have other ways of monitoring RabbitMQ, we don't necessarily have them in all cases).
So, to avoid having spurious error messages logged to Sentry, we can use a longer connection delay.
from deploy.
Related Issues (20)
- Make Docker stop before RabbitMQ, if possible
- mod_md errors
- OCDS 1.2 redirects
- Configure DKIM for Postfix, or update SPF HOT 3
- RabbitMQ: "All stable feature flags must be enabled after completing an upgrade" HOT 2
- Salt 3008 migration
- Upgrade ocp13 (registry) to Ubuntu 24.04 LTS HOT 4
- The container name "/data-registry-cron" is already in use by container HOT 1
- Consider Grafana for System Metrics HOT 3
- Hetzner DNS resolution errors HOT 2
- delete-after-x-days.sh file
- gateway4 and gateway6 are deprecated HOT 1
- PostgreSQL 12 End of Life on ocp13 HOT 1
- Remove Redash
- Dockerize redmine for local consultation
- Restore synchronous_standby_names and synchronous_commit defaults? (ocp23)
- Document --force parameter for local loads for non existing spiders
- Regular Docker clean-up
- Configure PostgreSQL monitoring for netdata HOT 5
- Update nr_hugepages on ocp13 and ocp23
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deploy.