An example to show how to include a faust project as a service using docker compose, with Kafka, Zookeeper and Schema Registry
Notice that everything runs using docker-compose, including the faust example application. For
local development is preferable to run the kafka cluster separate from the faust app.
If you want to generate a faust project from scratch, please use the cookiecutter-faust
Read more about Faust here: https://github.com/robinhood/faust
The project skeleton is defined as a medium/large project according to faust layout
The setup.py has the entrypoint to resolve the entrypoint problem
- Page Views: This application corresponds to Tutorial: Count page views
- Leader Election: This application corresponds to Tutorial: Leader Election
- Users: This is a custom application to demostrate how to integrate
FaustwithAvro Schema.
The Dockerfile is based on python:3.7-slim. The most important here is that the entrypoint will wait for kafka too be ready and after that execute the script run.sh
docker-compose.yaml includes zookeepeer, kafka and schema-registry based on confluent-inc.
For more information you can go to confluentinc and see the docker compose example here
Useful ENVIRONMENT variables that you may change:
| Variable | description | example |
|---|---|---|
| WORKER | Entrypoint in setup.py | example |
| WORKER_PORT | Worker port | 6066 |
| KAFKA_BOOSTRAP_SERVER | Kafka servers | kafka://kafka:9092 |
| KAFKA_BOOSTRAP_SERVER_NAME | Kafka server name | kafka |
| KAFKA_BOOSTRAP_SERVER_PORT | Kafka server port | 9092 |
| SCHEMA_REGISTRY_SERVER | Schema registry server name | schema-registry |
| SCHEMA_REGISTRY_SERVER_PORT | Schema registry server port | 8081 |
| SCHEMA_REGISTRY_URL | Schema Registry Server url | http://schema-registry:8081 |
- Start application:
make run-dev. This command start both the Page Views and Leader Election applications - Stop and remove containers:
make clean - List topics:
make list-topics - Send events to page_view topic/agent:
make send-page-view-event payload='{"id": "foo", "user": "bar"}'
Because we want to be sure that the message that we encode are valid we use Avro Schemas. Avro is used to define the data schema for a record's value. This schema describes the fields allowed in the value, along with their data types.
For our demostration in the Users application we are using the following schema:
{
"type": "record",
"namespace": "com.example",
"name": "AvroUsers",
"fields": [
{"name": "first_name", "type": "string"},
{"name": "last_name", "type": "string"}
]
}In order to use avro schemas with Faust we need to define a custom codec, a custom serializer and be able to talk with the schema-registry.
You can find the custom codec called avro_users registered using the codec registation approach described by faust.
The AvroSerializer is in charge to encode and decode messages using the schema registry client.
Now the final step is to integrate the faust model with the AvroSerializer.
# users.models
class UserModel(faust.Record, serializer='avro_users'):
first_name: str
last_name: strNow our application is able to send and receive message using arvo schemas!!!! :-)
Run tests with tox. Make sure that you have installed it.
tox- Application examples
- Integration with Schma Registry
- Schema Registry Client
- Custom codecs
- Custom Serializers
- Avro Schemas
- Make Schema Registry Client and Serializers a python package
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent