real-digital / esque-wire Goto Github PK

Kafka has implemented a feature for so called "tagged fields".
Those are supposed to reduce the amount of data that needs to be sent between client and server and to allow for more dynamic schema evolution.
With these fields it is possible to request or provide only specific fields/data and thereby eliminating the need to send unimportant information just to comply with the protocol.
Unfortunately tagged fields are currently not well documented at http://kafka.apache.org/protocol
Some more information can be found here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-482%3A+The+Kafka+Protocol+should+Support+Optional+Tagged+Fields

See: https://docs.python.org/3.6/library/asyncio-protocol.html#asyncio-protocol
This could make a few implementations a lot easier.

We only have a BrokerConnection class. We should create a ClusterConnection class as well, which is able to resolve and wrap all brokers in a cluster given some bootstrap servers. It should proably also inherit from BaseConnection and implement the send method.

Due to the way the packages and imports are designed at the moment, whenever you import esque_wire you will import all serializers and all data structures even though you might not need all of them. This leaves a very bad memory footprint and needs refactoring.

The BrokerConnection does not support SSL connections yet. We should add that functionality.

We should also support SASL authentication, ideally with pluggable classes like in https://github.com/real-digital/pykafka/blob/feature/sasl-scram-support/pykafka/sasl_authenticators.py

With Kafka Version 2.5.0 you get the following exception:

ValueError: 45 is not a valid ApiKey

This is because the enumSerializer that is used to deserialize the API key doesn't allow unknown enums and when the current ApiKey enum was generated for an older Kafka version, that API didn't exist yet.
Unknown API keys should be ignored.

Currently the primitive type RECORDS is the same as NULLABLE_BYTES. We should create a proper serializer for that.