Not all partitions are processed about kaffe HOT 9 CLOSED

@athal7 Thanks!

We've not seen this issue ourselves. I wonder if anyone else has. Most of our apps listen to multiple topics, but most (but not all) also specify the max_bytes parameter.

You mention lowering it: was it set but set to the same as the default (1,000,000)?

from kaffe.

from kaffe.

Not at all, just clarifying.

from kaffe.

I'm seeing this happen again, even with the max bytes specification:

$ /opt/kafka/bin/kafka-run-class.sh kafka.admin.ConsumerGroupCommand --bootstrap-server [redacted] --new-consumer --group mygroup --describe | grep mytopic

mytopic           0          -               9782            -          'mygroup@somehost'                 'mygroup@somehost'
mytopic           4          -               9076            -          'mygroup@somehost'                 'mygroup@somehost'
mytopic           8          -               9078            -          'mygroup@somehost'                'mygroup@somehost'
mytopic           1          -               9074            -          'mygroup@somehost'                 'mygroup@somehost'
mytopic           5          -               9076            -          'mygroup@somehost'                 'mygroup@somehost'
mytopic           9          -               9076            -          'mygroup@somehost'                 'mygroup@somehost'
mytopic           2          -               9076            -          'mygroup@somehost'                 'mygroup@somehost'
mytopic           6          9079            9079            0          'mygroup@somehost'                 'mygroup@somehost'
mytopic           3          -               9075            -          'mygroup@somehost'                  'mygroup@somehost'
mytopic           7          -               9075            -          'mygroup@somehost'                  'mygroup@somehost'

from kaffe.

oddly it eventually processed, about 30 minutes after sending this

from kaffe.

@athal7 It would be interesting to try to distill this down to a simple test so we could pinpoint the problem: Kaffe, Brod, Kafka, app code ... would be great to get to the bottom of this!

I wonder if you could try to reproduce it in a unit test.

from kaffe.

@objectuser agreed that that would be useful, curious if you have any ideas on how to go about that? It seems like one of those problems to me that is dependent on timing and amount of data, but could be wrong.

from kaffe.

It does not seem easy!

My only thought is to run batches of known data to a topic and read from it, noting how long it takes to consume the full set. If we can at least reproduce it after X iterations, maybe we can start bisecting the code to see where it all happens.

Maybe each batch could be a set of random numbers (enough for 10 or 100 keys per partition). Those could be sent through and the consumer could collect the data until all data is accounted for.

What do you think?

from kaffe.

We haven't seen this recently, closing.

from kaffe.