df-meilisearch-loader's Introduction

Meilisearch Loader

Application to load data from a Kafka topic and push the data to Meilisearch

The following table lists the environment variables that can be set, along with their default values.

Name	Description	Default
KAFKA_BROKER_HOST	The broker address used to connect to the kafka cluster	localhost:9092
KAFKA_TOPIC	The topic to consume the data from	test-topic
SCHEMA_REGISTRY_URL	The URL of the schema registry	localhost:8081
KAFKA_SASL_MECHANISM	SASL mechanisms. Valid options are `SCRAM-SHA-256`, `SCRAM-SHA-512`.	Blank
KAFKA_CLIENT_USERNAME	Username used for auth	Blank
KAFKA_CLIENT_SECRET	Secret used for auth	Blank
MEILISEARCH_URL	The address of the server Meilisearch is listening on	localhost:7700
MEILISEARCH_INDEX	The uid where documents will be grouped	test-index
MEILISEARCH_INSERT_BATCH_SIZE	The number of documents to batch before adding to the index	50
MEILISEARCH_API_KEY	Protects against unauthorized requests	aSampleMasterKey
MEILISEARCH_INDEX_PRIMARY_KEY	Uniquely identifies each document in an index	id

df-meilisearch-loader's People

Contributors

Currently, the loader starts a direct consumer, consuming from the earliest offset every time it comes up.

This results in:

duplicate reads from the topic when restarted (rereading from the beginning of the topic).
prevents horizontal scaling, since each instance would also result in duplicate reading