Repository consists of:

• Kafka all-in-one docker-compose script (based on cp-all-in-one)

• Kafka all-in-one helm charts (based on cp-helm-charts)

• Kafka helm charts for AWS MSK (based on cp-helm-charts)

Docker-Compose is a local dev environment. Including a postgres module to generate testing data in the docker-compose network.

./data-mock.py is an example script to generate demo data for testing.

Kubernetes charts is in ./helm/kafka, which is based on cp-helm-charts.

It can easily setup by helm:

cd helm

kubectl create ns kafka
helm install -f kafka/values-dev.yaml -n kafka kafka kafka

# patching existing helm
helm upgrade -f kafka/values-dev.yaml -n kafka kafka kafka

For production deployment on MSK, we should dump the values-dev.yaml to values-prod.yaml for another set of configurations.

cd helm

kubectl create ns kafka
helm install -f kafka/values-prod.yaml -n kafka kafka kafka

# patching existing helm
helm upgrade -f kafka/values-prod.yaml -n kafka kafka kafka

Here is the Kafka stack components and the sub-chart templates for corresponding kubernetes components:

Confluent Kafka components enable Java Management Extensions (JMX) for monitoring. By enabling ${component}.prometheus.jmx.enabled, components in kubernetes open port for exporting application metrics.

One can easily setup Prometheus monitoring in kubernetes with Prometheus Operator.