Comments (7)
I'm re-opning the issue as I see the same behavior on another environment.
This test env is running Docker For AWS (17.12.1-ce).
After several starts / stops of the swarmprom stack, the Caddy container is hanging and cannot be removed thus preventing the network from being removed as well.
Already noticed the same behavior ?
from swarmprom.
Hi @lucj can you please check Prometheus UI and see if that node is reachable?
from swarmprom.
In my current config, I have a nginx TLS termination in front of Caddy where only port 3000 is proxied. If I proxy the other ports (9090, 9093, 9094), I do not manage to have the whole stack working correctly.
I don't know if this is related but I see several errors like
"[ERROR 502 /] dial tcp: lookup grafana on 127.0.0.11:53: no such host" in Caddy.
from swarmprom.
ok, I activated ssl in Caddy instead of another nginx in front.
The prometheus console only shows 2 nodes
But Swarm has 3 nodes, each one is working fine.
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
ugk...909luj * xxx.compute.internal Ready Active Reachable 17.12.1-ce
ms7...yobt1n yyy.compute.internal Ready Active Leader 17.12.1-ce
mcd...lb6zpq zzz.compute.internal Ready Active Reachable 17.12.1-ce
from swarmprom.
Definitely not linked to swarmprom.
The problem seems to be linked to a connectivity issue in the mon_net network.
I guess it got lost after too many start/stop.
from swarmprom.
@stefanprodan just for the record: the reason why the 3rd node was not seen is because the mon_net network remained on this particular node after the stack was removed (this is a known bug moby/moby#35204).
When the stack was started once again, the mon_net network was recreated with the same subnet thus preventing the node-exporter to attach to the network as the subnet is already used.
c61dob322u72hskjyk5nn0dzg mon_node-exporter.ms71f27l8jwtqexhr53yobt1n stefanprodan/swarmprom-node-exporter:v0.15.2@sha256:0575845ee924fa91138804663a12207ed53a56542d257273ffb9b30e22b78cd1 ip-172-31-27-219.eu-west-1.compute.internal Ready Rejected 3 seconds ago "failed to allocate gateway (10.0.0.1): Address already in use"
zxv4pi94uii1gmfok9c9b28mp \_ mon_node-exporter.ms71f27l8jwtqexhr53yobt1n
...
The thing is I cannot remove the network from this particular node as a container is still connected to it and for an unknown reason refused to be removed as well (!).
~ $ docker network rm mon_net
Error response from daemon: network mon_net id 1d34qanxyqqxaytteudtd1pfj has active endpoints
~ $ docker network inspect mon_net
...
"Containers": {
"80c0fbb70da2342b82367897ddd1028593f70a87b2b5929804b4ee527c10b12f": {
"Name": "mon_caddy.1.vrm2namdg5vc2mk9fqcdbk1xs",
"EndpointID": "c3d1aeb8b3089a51e0ec3c05703b7c6d81e2a9f95238a6aa1291f38e1c84bdb3",
"MacAddress": "02:42:0a:00:00:16",
"IPv4Address": "10.0.0.22/24",
"IPv6Address": ""
}
},
...
~ $ docker container rm -f 80c0fbb70da2342b
=> hanging....
I then specified another network with it's own subnet and it's working fine.
networks:
netx:
driver: overlay
attachable: true
ipam:
driver: default
config:
- subnet: 10.0.20.0/24
from swarmprom.
I am already facing the same issue
from swarmprom.
Related Issues (20)
- How do I clean the history of prometheus? HOT 1
- Updating packages HOT 2
- Docker compose empty environment variables HOT 1
- Grafana Loki integration
- Running swarmprom behind traefik HOT 3
- Docker swarm node - no data showing HOT 1
- Grafana & Caddy service is instantly terminating HOT 3
- Docker Swarm Nodes dashboard empty HOT 1
- Node exporter not starting through docker service create HOT 1
- add blackbox_exporter
- node_meta only contains mapping for 1 host HOT 1
- Mail notification HOT 1
- Mountpoint set to "/", swap file returns NaN and upgrade of containers ? HOT 1
- worker node details are missing in docker swarm node dashboard HOT 1
- swarm_node.rules.yml: no such file or directory
- Replace Unsee with it's successor Karma
- Swarmprom for kubernetes
- node exporter sends whole nodes metrics
- Custom Grafana Dashboards on Startup
- About project future HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from swarmprom.