Adding health check for Zookeeper and Kafka pods about strimzi-kafka-operator HOT 2 CLOSED

I fixed the Zookeeper healthcheck. That seemed to be quite easy and there seems to be only single solution used by almost everyone.

Kafka doesn't seem to have any obvious solutions. I saw several solutions using bin/kafka-broker-api-versions.sh. Do you think that is a good option? It seems better to me than just pinging the TCP port.

WDYT?

from strimzi-kafka-operator.

The broker doesn't have to do much to return ApiVersionsResponse (it's basically just a static method), but it would test things a little more than just doing a TCP handshake. It would report OK when the broker was an a bad state e.g. unable to write records to disk.

Internally the KafkaHealthCheck uses an ephermal znode in zookeeper, and this is how other brokers know about the health of a broker, but it seems to me this would suffer from the same problem.

A much more thorough test would be where we test producing and consuming to a compacted topic with one partition for each broker, and the partition wasn't replicated (so we knew only a single broker could be handling the produce and fetch requests). This would test something much closer to what producers and consumers actually care about. However using standard tools this would require a metadata request to a bootstrap broker. Perhaps we could write a custom tool to do this.

I would be inclined to implement bin/kafka-broker-api-versions.sh for the time being and do something better when we know more about situations where it is inadequate.

from strimzi-kafka-operator.