Currently, KAFKA_SOURCE_TOPICS and KAFKA_RETRY_INTERVALS env vars, amongst some others, control how this module behaves. The problem with this is that all topics provided as a comma-separated string, and the implementing application needs to provide a HandlerMap where each key is the topic name that is being consumed.

This means that applications either need to hard-code their topic names in their HandlerMap, or duplicate topic names into their own environment variables to manage them externally.

If we removed the KAFKA_* env vars as a way of configuring this module, then we could update config.NewConfig() to accept these options instead, and then the implementing application can decide how to manage this configuration externally.

When a handler returns an error for a message, the consumer triggers the publishing to retry queues with only the payload and misses adding the headers and key (latter used for partitioning sometimes).

Currently you can only assign handlers to topics, but it seems common in Kafka to have many message types in a single topic, this leads to each Handler having to know about several message types and needing to delegate conditionally.

We could possibly introduce the possiblity be more opinonated and assign Handlers based on topic AND type.

Currently, we rely on unit testing to prove that this module works, but that is not enough. We should have a full integration test suite that simulates the use of this module to consume messages from a Kafka broker.

The effort to use this module in the implementing service is quite large, as demonstrated in https://github.com/inviqa/kafka-consumer-go/blob/master/tools/docs/implementing-a-handler.md.

It would be nicer to simplify the setup, importantly we should wrap up the creation of the producer and consumer collection together.

If we can find any other simplifications then we should make those, and not worry about breaking the API right now as we have not tagged anything up.

Currently when using DB retries, this is only configured to use Postgres, due to the driver being hardcoded on the following line. This should be dynamic so it can work with both Postgres and MySQL

https://github.com/inviqa/kafka-consumer-go/blob/master/config/config.go#L165

Currently, when we store messages for retry, they are sent back to Kafka and stored in a retry topic. Dead-lettered messages are also stored in a topic.

This limits us in terms of how long we can delay consumption of messages, and it's also not an exact science in terms of how long a message is delayed for before it gets retried, because we only sleep in the consumer between consume calls and therefore it can be far longer for messages to be retried than originally intended.

We should add the option of using a DB for retries and dead-lettered messages, with a separate goroutine running if this option is enabled (instead of the retry/dead-letter topic consumer).

Eventually, this should become the only way of handling retries as it is far more precise and cheaper.

Tasks:

• Add database storage retry mechanism (done in #16)
• Add job runners to allow apps to easily run maintenance cleanup on the retry table (see #18)