Kafka Connect Demo
The purpose of this repository is to :
- provide a slideshow that explains how Kafka Connect can be used, its principles and guidelines to set it up to connect Kafka and Psql
- A set of script based on Docker images and shell scripts to let people run the demo by themselves
- Show some use cases
- Download git
- Get started with docker
- Install docker-compose
curl
command available or a REST client like Postman / Insomnia
In this way to use kafka-connect, we will deploy the entire stack on docker environnement. Here we use the Confluent elements :
schema-registry
- "It provides a RESTful interface for storing and retrieving your Avro, JSON Schema, and Protobuf schemas."kafka-connect
- "Kafka Connect is a tool for scalably and reliably streaming data between Apache Kafka and other data systems."
See also Kafka Connect Tutorial on Docker
kafka-connect
need plugins to interact with databases, download them and extract the content in the plugins
folder :
Kafka Connect JDBC
- "JDBC source and sink connectors"Debezium PostgreSQL CDC Connector
- "Debezium’s PostgreSQL Connector can monitor and record the row-level changes in the schemas of a PostgreSQL database"
See also Discover Kafka connectors and more
Now, we can launch the environment :
$ docker-compose --project-name kafka-connect-pgsql -f kafka-connect.yml up -d
Creating postgres-sink ... done
Creating postgres-source ... done
Creating zookeeper ... done
Creating kafka ... done
Creating confluent-registry ... done
Creating confluent-connect ... done
Check if confluent-connect
container is up and ready :
$ docker logs confluent-connect | grep started
...
[2021-01-08 00:32:41,837] INFO REST resources initialized; server is started and ready to handle requests (org.apache.kafka.connect.runtime.rest.RestServer)
[2021-01-08 00:32:41,837] INFO Kafka Connect started (org.apache.kafka.connect.runtime.Connect)
For create connectors to extract/inject datas between Kafka and external databases, we utilize curl commands trough the api kafka-connect
endpoints :
source
- Database to Kafka
curl -X POST http://localhost:8083/connectors \
-H "Content-Type: application/json" \
--data @connectors/postgresql-source.json
We define in postgresql-source.json file the connection informations and the elements to watch (the table employees in this case). All updated rows will send on topic hrdata.public.employees
{
"name": "postgresql-source-connector",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"database.hostname": "postgres-source",
"database.port": "5432",
"database.user": "user",
"database.password": "password",
"database.dbname" : "db",
"database.server.name": "hrdata",
"table.include.list": "public.employees",
...
}
}
sink
- Kafka to Database
curl -X POST http://localhost:8083/connectors \
-H "Content-Type: application/json" \
--data @connectors/postgresql-sink.json
Like the source file configuration, it exists the same for the sink side postgresql-sink.json. If not exists, the target table will be create.
{
"name": "postgresql-sink-connector",
"config": {
"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"connection.url": "jdbc:postgresql://postgres-sink:5432/db?user=user&password=password",
"topics": "hrdata.public.employees",
"table.name.format": "employees",
"insert.mode": "insert",
"auto.create": true,
}
}
See more endpoints on Connect REST Interface documentation
Let's try it out ! Update or insert rows on public.employees
table in postgres-source database.
See the differents ways to explore databases of this project
See an use case example
$ docker-compose --project-name kafka-connect-pgsql -f kafka-connect.yml stop
Stopping confluent-connect ... done
Stopping confluent-registry ... done
Stopping kafka ... done
Stopping postgres-source ... done
Stopping zookeeper ... done
Stopping postgres-sink ... done
$ docker-compose --project-name kafka-connect-pgsql -f kafka-connect.yml rm
Going to remove postgres, connect, kafka, zookeeper
Are you sure? [yN] y
Removing confluent-connect ... done
Removing confluent-registry ... done
Removing kafka ... done
Removing postgres-source ... done
Removing zookeeper ... done
Removing postgres-sink ... done
docker run --net=host --rm \
wurstmeister/kafka:2.11-2.0.0 sh opt/kafka_2.11-2.0.0/bin/kafka-topics.sh \
--zookeeper localhost:2181 \
--list
docker run --net=host --rm \
wurstmeister/kafka:2.11-2.0.0 sh opt/kafka_2.11-2.0.0/bin/kafka-console-consumer.sh \
--bootstrap-server localhost:9092 \
--topic hrdata.public.employees \
--timeout-ms 3000 \
--from-beginning
docker run --net=host --rm \
confluentinc/cp-schema-registry:5.0.0 kafka-avro-console-consumer \
--bootstrap-server localhost:9092 \
--topic hrdata.public.employees \
--timeout-ms 3000 \
--from-beginning \
- Kafka Connect Tutorial on Docker
- Schema Management Overview
- Kafka Connect
- JDBC Source Connector for Confluent Platform
- JDBC Sink Connector for Confluent Platform
- Discover Kafka connectors and more
- Connect REST Interface
- PostgreSQL Source Connector (Debezium) Configuration Properties