Comments (5)
Troubleshooting further it seems my issue stems from an unclean shutdown of the graylog docker container - mongodb and opensearch shutdown correctly so don't trigger the restart on-failure condition, but graylog apparently shutdown cleanly.
My operating system (Manjaro) shuts down very quickly, and is probaby not allowing enough time. I've since added graceful shutdown to all 3 containers but that hasn't helped. I'm now crawling through logs trying to find clues as to why graylog is crashing, or sending the wrong signal, or whether Manjaro is just not waiting for an all-clear.
from docker-compose.
Here are the logs from shutdown for graylog (from the operating system logs) - nothing jumps out at me:
20:05:18 dockerd: time="2023-06-26T20:05:18.916975194+10:00" level=warning msg="ShouldRestart failed, container will not be restarted" container=ade8dd9f6c3cdc014db073546cd1e195e781748e6dc9b42a3fb7322419e0fa60 daemonShuttingDown=true error="restart canceled" execDuration=19m30.178415309s exitStatus="{143 2023-06-26 10:05:17.589297607 +0000 UTC}" hasBeenManuallyStopped=false restartCount=0
20:05:18 dockerd: time="2023-06-26T20:05:18.916975194+10:00" level=warning msg="ShouldRestart failed, container will not be restarted" container=ade8dd9f6c3cdc014db073546cd1e195e781748e6dc9b42a3fb7322419e0fa60 daemonShuttingDown=true error="restart canceled" execDuration=19m30.178415309s exitStatus="{143 2023-06-26 10:05:17.589297607 +0000 UTC}" hasBeenManuallyStopped=false restartCount=0
20:05:18 dockerd: time="2023-06-26T20:05:18.899268741+10:00" level=info msg="ignoring event" container=ade8dd9f6c3cdc014db073546cd1e195e781748e6dc9b42a3fb7322419e0fa60 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
20:05:18 containerd: time="2023-06-26T20:05:18.899190691+10:00" level=warning msg="cleaning up after shim disconnected" id=ade8dd9f6c3cdc014db073546cd1e195e781748e6dc9b42a3fb7322419e0fa60 namespace=moby
20:05:18 containerd: time="2023-06-26T20:05:18.899190691+10:00" level=warning msg="cleaning up after shim disconnected" id=ade8dd9f6c3cdc014db073546cd1e195e781748e6dc9b42a3fb7322419e0fa60 namespace=moby
20:05:18 containerd: time="2023-06-26T20:05:18.899070267+10:00" level=info msg="shim disconnected" id=ade8dd9f6c3cdc014db073546cd1e195e781748e6dc9b42a3fb7322419e0fa60 namespace=moby
20:05:18 systemd: docker-ade8dd9f6c3cdc014db073546cd1e195e781748e6dc9b42a3fb7322419e0fa60.scope: Consumed 1min 53.928s CPU time.
from docker-compose.
here are the last few lines of the docker logs - does this look like a successful shutdown?
{"log":"\u0009at org.graylog2.database.PersistedServiceImpl.save(PersistedServiceImpl.java:198)\n","stream":"stderr","time":"2023-06-26T10:05:17.031969692Z"}
{"log":"\u0009at org.graylog2.system.activities.SystemMessageActivityWriter.write(SystemMessageActivityWriter.java:56)\n","stream":"stderr","time":"2023-06-26T10:05:17.031975488Z"}
{"log":"\u0009at org.graylog2.commands.Server$ShutdownHook.run(Server.java:326)\n","stream":"stderr","time":"2023-06-26T10:05:17.031980878Z"}
{"log":"\u0009at java.base/java.lang.Thread.run(Unknown Source)\n","stream":"stderr","time":"2023-06-26T10:05:17.031986271Z"}
from docker-compose.
I'm a little out of my depth here, but it looks like the java graceful shutdown issue that has popped up a few times before:
Graylog2/graylog-docker#173
Graylog2/graylog-docker#87
from docker-compose.
Okay so I think there are 2 issues:
- it looks like the "jave not shutting down gracefully" bug is back again, so an issue for the docker container repository, not the docker-compose repository, and
- docker has some short-comings - the docker engine doesn't know about depends_on values in docker-compose, so in cases like the above although graylog will restart because it failed, it won't trigger the depends_on condition. Using "unles-stopped" on all containers brought them all up after a restart to my suprise as I thought it would have counted mongo and opensearch as having been stopped.
from docker-compose.
Related Issues (20)
- The connection does not work after opensearch is replaced with datanode. HOT 19
- Error on Data-Node after provision certificate HOT 4
- Opensearch is not available after provision certificate HOT 10
- Email configuration doesn't apply HOT 1
- OpenSearch/Elasticsearch is not available. HOT 9
- Introduction of datanode may be confusing for users, possibly need to add text in the docker compose file to communication how to perform first setup HOT 10
- confusing documentation HOT 2
- SSL Certificate Hostname Verification Issue with Graylog in Docker Environment HOT 3
- The graylog/graylog-datanode:5.2 docker image is not starting correctly returning an error max_map_count value should be at least 262144 but is 65530 HOT 3
- Cannot connect to data node HOT 1
- datanode won't restart after hitting flood-stage watermark HOT 13
- DataNode: Failed to load keystore from Mongo collection for node GRAYLOG CA HOT 2
- Node API Browser uses Docker internal IP HOT 1
- The signing key's size is 208 bits which is not secure enough HOT 4
- How to connect to Data Node to fix long messages not logged HOT 5
- Facing error org.graylog2.bootstrap.CmdLineTool - Startup error HOT 1
- certificates expired for datanode after long downtime HOT 5
- How to generate password and SHA2 in Windows? HOT 1
- docker compse max_map_count value should be at least 262144 but is 65530 HOT 2
- Add to readme which folder may be used for single node use HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from docker-compose.