A Python implementation of "Kafka Connect"-like functionality that uses plain AvroConsumer/-Producers to make it easier to develop Kafka connect applications.
PyConnectSources should be able to share a kafka topic so we do not have to create a new topic for each source. Sources will have to get an ID similar to the consumer group id which will be used to key their offset within the topic.
The ID can also be used as consumer group id by the consumer that reads from that offset topic so it won't have to go through the whole topic history.
In order to further cut down seek time, we can also initialize the topic with a rather high amount of partitions (say 50) and enable compaction on it.
Having examples that aren't working is embarrassing and we should try to avoid that.
The tests for the examples were disabled in #19, we need to re enable them, in order to make sure the examples don't break.
At the moment the source connector infers key and value schemas on first write. We should force subclasses to implement their own schema generation and expose the instance attributes holding the schemas so they can be overwritten on demand.
Librdkafka changed the default for enable.partition.eof from True to False. We didn't set that explicitly and were relying on the default value. We need to change that.
We get a lot of 2019-05-23 12:53:38,831: INFO - pyconnect.pyconnectsink - Committing offsets: []
messages in our sink in case no data comes in. This might happen because when a connector restarts and never gets a message, it can't get the correct offset to commit. This might be the cause of our sinks getting reset.
In order to keep our code boundaries clean, we should wrap third party libraries and provide our own interfaces which are tailored to our needs.
This makes it easier to test our code or to switch to another library if necessary.
We should also introduce tests for third party APIs in order to make sure they behave the way we expect them to.