real-digital / pyconnect Goto Github PK

In order to clean up the code and make testing easier, we should move handling of the producer offsets into a separate class.

PyConnectSources should be able to share a kafka topic so we do not have to create a new topic for each source. Sources will have to get an ID similar to the consumer group id which will be used to key their offset within the topic.
The ID can also be used as consumer group id by the consumer that reads from that offset topic so it won't have to go through the whole topic history.
In order to further cut down seek time, we can also initialize the topic with a rather high amount of partitions (say 50) and enable compaction on it.

Make sure we can gracefully shut down if the system requests it.

Please see https://www.confluent.io/blog/kafka-listeners-explained/

Having examples that aren't working is embarrassing and we should try to avoid that.
The tests for the examples were disabled in #19, we need to re enable them, in order to make sure the examples don't break.

Configure logging to use a json format, compatible with DataDog.

At the moment the source connector infers key and value schemas on first write. We should force subclasses to implement their own schema generation and expose the instance attributes holding the schemas so they can be overwritten on demand.

All tests should be able to run both with a real kafka backend and with a mock. We wouldn't want to maintain two different sets of tests.

Librdkafka changed the default for enable.partition.eof from True to False. We didn't set that explicitly and were relying on the default value. We need to change that.

We get a lot of
2019-05-23 12:53:38,831: INFO - pyconnect.pyconnectsink - Committing offsets: []
messages in our sink in case no data comes in. This might happen because when a connector restarts and never gets a message, it can't get the correct offset to commit. This might be the cause of our sinks getting reset.

probably enough to update self.__offsets during _on_assign here https://github.com/real-digital/pyconnect/blob/master/src/pyconnect/pyconnectsink.py#L175

In order to keep our code boundaries clean, we should wrap third party libraries and provide our own interfaces which are tailored to our needs.
This makes it easier to test our code or to switch to another library if necessary.
We should also introduce tests for third party APIs in order to make sure they behave the way we expect them to.