tamediadigital / kafka-d Goto Github PK
View Code? Open in Web Editor NEWkafka-d is a D Kafka client that depends on the vibe.d framework.
License: MIT License
kafka-d is a D Kafka client that depends on the vibe.d framework.
License: MIT License
I've been taking a look at the source in consumer.d within processBuffer. It seems kafka-d only handles the older message format (prior to Kafka 0.11) which are structured like:
MessageSet (Version: 0) => [offset message_size message]
offset => INT64
message_size => INT32
message => crc magic_byte attributes key value
crc => INT32
magic_byte => INT8
attributes => INT8
bit 0~2:
0: no compression
1: gzip
2: snappy
bit 3~7: unused
key => BYTES
value => BYTES
newer versions of Kafka use the following structure which includes a timestamp as well as also supporting lz4 compression.
MessageSet (Version: 1) => [offset message_size message]
offset => INT64
message_size => INT32
message => crc magic_byte attributes timestamp key value
crc => INT32
magic_byte => INT8
attributes => INT8
bit 0~2:
0: no compression
1: gzip
2: snappy
3: lz4
bit 3: timestampType
0: create time
1: log append time
bit 4~7: unused
timestamp => INT64
key => BYTES
value => BYTES
source/kafkad/consumer/consumer.d(21): Error: cannot create a struct until its size is determined
source/kafkad/consumer/consumer.d(22): Error: cannot create a struct until its size is determined
When integrating with Kafka using Spring Framework it's easy to get a producer/consumer running without needing to configure anything and when running a producer, if the topic doesn't already exist, Spring will just create it. It's also configurable via auto.create.topics.enable
which defaults to true.
It would be great if kafka-d could do the same as it really simplifies deploying a new producer
vibe-d/vibe.d#1345
makes kafka-d not work anymore
Didn't find any possibility to specify consumer group id for connection to use Kafka consumer message share mode (see https://www.tutorialspoint.com/apache_kafka/apache_kafka_workflow.htm, Workflow of Queue Messaging / Consumer Group)
Hope I missed something?
https://github.com/tamediadigital/kafka-d/blob/master/source/kafkad/connection.d#L259
probably because of the mutex
fix all module/package visibility warnings
0.16.0 beta is out: http://forum.dlang.org/post/[email protected]
todo:
submodules/kafka-d/source/kafkad/exception.d(7): Error: @nogc function 'kafkad.exception.ConnectionException.ExceptionCtorMixin!().this' cannot call non-@nogc function 'object.Exception.this'
submodules/kafka-d/source/kafkad/exception.d(43): Error: function kafkad.exception.rethrow!(ConnectionException, NetworkAddress, string).rethrow no return exp; or assert(0); at end of function
submodules/kafka-d/source/kafkad/client.d(148): Error: template instance kafkad.exception.rethrow!(ConnectionException, NetworkAddress, string) error instantiating
submodules/kafka-d/source/kafkad/exception.d(7): Error: @nogc function 'kafkad.exception.ProtocolException.ExceptionCtorMixin!().this' cannot call non-@nogc function 'object.Exception.this'
submodules/kafka-d/source/kafkad/exception.d(7): Error: @nogc function 'kafkad.exception.CrcException.ExceptionCtorMixin!().this' cannot call non-@nogc function 'object.Exception.this'
submodules/kafka-d/source/kafkad/consumer.d(78): Error: function kafkad.consumer.Consumer.getMessage no return exp; or assert(0); at end of function
submodules/kafka-d/source/kafkad/exception.d(10): Error: @nogc function 'kafkad.exception.ConnectionException.ExceptionCtorMixin!().this' cannot call non-@nogc function 'object.Exception.this'
submodules/kafka-d/source/kafkad/exception.d(7): Error: @nogc function 'kafkad.exception.StreamException.ExceptionCtorMixin!().this' cannot call non-@nogc function 'object.Exception.this'
submodules/kafka-d/source/kafkad/exception.d(10): Error: @nogc function 'kafkad.exception.StreamException.ExceptionCtorMixin!().this' cannot call non-@nogc function 'object.Exception.this'
submodules/kafka-d/source/kafkad/exception.d(7): Error: @nogc function 'kafkad.exception.MetadataException.ExceptionCtorMixin!().this' cannot call non-@nogc function 'object.Exception.this'
submodules/kafka-d/source/kafkad/exception.d(10): Error: @nogc function 'kafkad.exception.MetadataException.ExceptionCtorMixin!().this' cannot call non-@nogc function 'object.Exception.this'
submodules/kafka-d/source/kafkad/exception.d(10): Error: @nogc function 'kafkad.exception.ProtocolException.ExceptionCtorMixin!().this' cannot call non-@nogc function 'object.Exception.this'
submodules/kafka-d/source/kafkad/exception.d(10): Error: @nogc function 'kafkad.exception.CrcException.ExceptionCtorMixin!().this' cannot call non-@nogc function 'object.Exception.this'
submodules/kafka-d/source/kafkad/protocol/deserializer.d(67): Error: unrecognized pragma(inline)
submodules/kafka-d/source/kafkad/protocol/serializer.d(35): Error: unrecognized pragma(inline)
implement an interface to submit a Partitioner similar to the java library
push message should support a few more types:
if the current broker reboots the Consumer crashes and cannot recover.
@pszturmaj any ideas?
if the producer goes down and/or the connection is capped there is a deadlock
at least here:
https://github.com/tamediadigital/kafka-d/blob/master/source/kafkad/producer.d#L218
but possible at other places aswell.
expected behaviour:
the client should not care, if the connection drops, kafka-d should re-establish it
add example to show how to recover from a
Offset 285960896 is out of range for topic mytopic, partition 0
Exception
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.