How to mirror data between Redpanda and Kafka clusters using Kafka Connect and MirrorMaker2. This demo simulates a hypothetical scenario where Redpanda is running in a supermarket to stream events generated by point-of-sale (pos) devices to a central Kafka cluster that aggregates events from all supermarkets.
docker-compose up -d
[+] Running 6/6
⠿ Network redpanda-mm2_default Created
# The Redpanda instance running in the supermarket
⠿ Container redpanda Started
# The Kafka, Zookeeper, and Kafka Connect instances running in the central data centre
⠿ Container zookeeper Started
⠿ Container kafka-dc Started
⠿ Container kafka-connect Started
⠿ Container redpanda-console Started
export REDPANDA_BROKERS=localhost:9092
export KAFKA_BROKERS=localhost:9992
# Create the point-of-sale "pos" topic in Redpanda
rpk topic create pos --brokers ${REDPANDA_BROKERS}
rpk topic list --brokers ${REDPANDA_BROKERS}
# Create the topic "prices" in Kafka
rpk topic create prices --brokers ${KAFKA_BROKERS}
rpk topic list --brokers ${KAFKA_BROKERS}
# Connector to mirror the topic "pos" from Redpanda to Kafka "shop.pos"
cat shop-connector.json | curl -X POST -H "Content-Type: application/json" http://localhost:8083/connectors --data-binary @- | jq .
# Connector to mirror the topic "prices" from Kafka to Redpanda "dc.prices"
cat dc-connector.json | curl -X POST -H "Content-Type: application/json" http://localhost:8083/connectors --data-binary @- | jq .
Check the connector status in Redpanda Console: http://localhost:8080/kafka-connect/kafka-connect
Generate random point-of-sale data in the supermarket (Redpanda) and consume it from the central cluster in the data centre (Kafka)
# Produce data to the "pos" topic in Redpanda
for i in {1..60}; do echo "$RANDOM" | rpk topic produce pos --brokers ${REDPANDA_BROKERS}; sleep 1; done
# Consume data from the "shop.pos" topic in Kafka
rpk topic consume shop.pos --brokers ${KAFKA_BROKERS}
Generate random price change events in the central cluster (Kafka) and consume them on the supermarket side (Redpanda). This shows that topics can be replicated in both directions.
# Produce data to the "prices" topic in Kafka
for i in {1..60}; do echo "$RANDOM" | rpk topic produce prices --brokers ${KAFKA_BROKERS}; sleep 1; done
# Consume data from the "dc.prices" topic in Redpanda
rpk topic consume dc.prices --brokers ${REDPANDA_BROKERS}
docker-compose down -v