Giter VIP home page Giter VIP logo

redpanda-mm2's Introduction

Redpanda + Kafka Connect + MirrorMaker2

How to mirror data between Redpanda and Kafka clusters using Kafka Connect and MirrorMaker2. This demo simulates a hypothetical scenario where Redpanda is running in a supermarket to stream events generated by point-of-sale (pos) devices to a central Kafka cluster that aggregates events from all supermarkets.

Start the containers

docker-compose up -d
[+] Running 6/6
 ⠿ Network redpanda-mm2_default  Created
 # The Redpanda instance running in the supermarket
 ⠿ Container redpanda            Started
 # The Kafka, Zookeeper, and Kafka Connect instances running in the central data centre
 ⠿ Container zookeeper           Started
 ⠿ Container kafka-dc            Started 
 ⠿ Container kafka-connect       Started 
 ⠿ Container redpanda-console    Started

Create environment variables

export REDPANDA_BROKERS=localhost:9092
export KAFKA_BROKERS=localhost:9992

Create the topics

# Create the point-of-sale "pos" topic in Redpanda
rpk topic create pos --brokers ${REDPANDA_BROKERS}
rpk topic list --brokers ${REDPANDA_BROKERS}

# Create the topic "prices" in Kafka
rpk topic create prices --brokers ${KAFKA_BROKERS}
rpk topic list --brokers ${KAFKA_BROKERS}

Deploy the connectors using the Connect REST API

# Connector to mirror the topic "pos" from Redpanda to Kafka "shop.pos"
cat shop-connector.json | curl -X POST -H "Content-Type: application/json" http://localhost:8083/connectors --data-binary @- | jq .

# Connector to mirror the topic "prices" from Kafka to Redpanda "dc.prices"
cat dc-connector.json | curl -X POST -H "Content-Type: application/json" http://localhost:8083/connectors --data-binary @- | jq .

Check the connector status in Redpanda Console: http://localhost:8080/kafka-connect/kafka-connect

Generate point-of-sale data

Generate random point-of-sale data in the supermarket (Redpanda) and consume it from the central cluster in the data centre (Kafka)

# Produce data to the "pos" topic in Redpanda
for i in {1..60}; do echo "$RANDOM" |  rpk topic produce pos --brokers ${REDPANDA_BROKERS}; sleep 1; done
# Consume data from the "shop.pos" topic in Kafka
rpk topic consume shop.pos --brokers ${KAFKA_BROKERS}

Generate pricing data

Generate random price change events in the central cluster (Kafka) and consume them on the supermarket side (Redpanda). This shows that topics can be replicated in both directions.

# Produce data to the "prices" topic in Kafka
for i in {1..60}; do echo "$RANDOM" |  rpk topic produce prices --brokers ${KAFKA_BROKERS}; sleep 1; done
# Consume data from the "dc.prices" topic in Redpanda
rpk topic consume dc.prices --brokers ${REDPANDA_BROKERS}

Clean up

docker-compose down -v

redpanda-mm2's People

Contributors

jrkinley avatar

Stargazers

 avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.