Monitor Missing/Crashing containers about swarmprom HOT 3 CLOSED

You can add alert rules in https://github.com/stefanprodan/swarmprom/blob/master/prometheus/rules/swarm_task.rules.yml and redeploy.

An alert would look like:

  - alert: kafka
    expr: count(container_memory_rss{container_label_com_docker_swarm_task_name=~".+kafka.+"})
      < 3
    for: 1m
    annotations:
      description: 'Kafka cluster down'
      summary: 'Kafka cluster nodes count < 3'

from swarmprom.

Hi Stefan,

Thanks for the reply.

Now when I put this conf and redeployed the stack, the count doubled to 6 and I 'm getting duplicate records.

For the same kafka instance I 'm getting two records. So is this method reliable?

container_memory_rss{container_label__base_version="x86_64-0.4.5",container_label__version="",container_label_com_docker_stack_namespace="kafka",container_label_com_docker_swarm_node_id="lftgekvut6idwdb60omqp612i",container_label_com_docker_swarm_service_id="i6ebvyb9zdtvecr2jvipfxubc",container_label_com_docker_swarm_service_name="kafka_kafka0",container_label_com_docker_swarm_task="",container_label_com_docker_swarm_task_id="py5vx3sye1ou9vxdqvelv6m1r",container_label_com_docker_swarm_task_name="kafka_kafka0.1.py5vx3sye1ou9vxdqvelv6m1r",id="/docker/5a781f09efdeb048a573ab4a3a6d14d56eb7261a93e8a86442e909c476397ee0",image="hyperledger/fabric-kafka:latest",instance="10.0.0.87:8080",job="cadvisor",name="kafka_kafka0.1.py5vx3sye1ou9vxdqvelv6m1r"}

268419072

container_memory_rss{container_label__base_version="x86_64-0.4.5",container_label__version="",container_label_com_docker_stack_namespace="kafka",container_label_com_docker_swarm_node_id="lftgekvut6idwdb60omqp612i",container_label_com_docker_swarm_service_id="i6ebvyb9zdtvecr2jvipfxubc",container_label_com_docker_swarm_service_name="kafka_kafka0",container_label_com_docker_swarm_task="",container_label_com_docker_swarm_task_id="py5vx3sye1ou9vxdqvelv6m1r",container_label_com_docker_swarm_task_name="kafka_kafka0.1.py5vx3sye1ou9vxdqvelv6m1r",id="/docker/5a781f09efdeb048a573ab4a3a6d14d56eb7261a93e8a86442e909c476397ee0",image="hyperledger/fabric-kafka:latest",instance="10.0.0.98:8080",job="cadvisor",name="kafka_kafka0.1.py5vx3sye1ou9vxdqvelv6m1r"}

from swarmprom.

It stabilized after a while.

from swarmprom.