razvan / kafka-s3-consumer Goto Github PK

I was running your code, but after some testing it just 'stopped-working'.

turns out that after inserting messages in the kafka-queue with length >kafka.maxmessagesize (the default is 256 bytes, the thing throws an exception:

Method threw 'kafka.common.InvalidMessageSizeException' exception.

at line 219: App.java

FetchRequest fetchRequest = new FetchRequest(topic, partition, offset, conf.getKafkaMaxMessageSize());
MessageSet messageSet = consumer.fetch(fetchRequest)
while (!messageSet.iterator().hasNext()) {
...


Any plans to update this consumer for 0.8.0?