Repeated rebalance cycle with kafka broker 2.3.0 about kaffe HOT 16 CLOSED

@dtykocki We tried with a Kafka 2.3.1 broker and the rebalance cycle didn't occur. Thank you for checking!
We will upgrade our brokers soon then :) I think this issue can be closed as the error is caused by the mentioned kafka bug.

from kaffe.

@objectuser
Unfortunately it was just coincidence yesterday, when I changed the config it continued to consume the topic but later it stopped again.

from kaffe.

@nemzsom I wonder if there's an incompatibility at a lower level. Can you try updating the brod dependency and see if that helps?

from kaffe.

@objectuser I tried out with the latest release from brod 3.8.1 but the issue still persist.

from kaffe.

Very interesting! We're using Kafka 2.0.1 hosted by Heroku and have not experienced those issues.

Are you self hosting?

from kaffe.

Do you have any other clusters? Wondering if you can see if it happens in new/isolated environments. Maybe create a cluster in Docker, etc.?

from kaffe.

Yes, we are self hosting and we encountered this issue after we upgrade to 2.3.0 from 1.0.1.

And it is reproducible with a single broker running in docker, no cluster needed.

from kaffe.

I've been able to reproduce this issue with Kafka 2.3.0, though it doesn't happen on the version we're using, 2.0.1. So that narrows the issue down a bit.

We'll investigate further.

from kaffe.

@nemzsom Looking at the changes in 2.3.0, I am thinking that the new "cooperative" rebalancing strategy might have something to do with this.

Would you be able to try setting connect.protocol=eager in your Kafka Connect config and see if it improves the rebalancing issue?

This might not be a long term fix, as that protocol is said to be deprecated. However, it might be a stop gap to address the issue until we can update Kaffe to handle that protocol.

EDIT: That looks like it's just for Kafka Connect, so maybe not relevant. 🤔

from kaffe.

Thank you @objectuser for looking into this!
I’m on vacation right now but I’ll try your suggestion when I’m back.
(Although I don’t know where to put the connect config so maybe it is only for kafka connect indeed)

from kaffe.

Colleague of @nemzsom here :)
Unfortunately the connect.protocol=eager seems not relevant in this case as we are not using kafka connect.

I found a somewhat relevant question on stackoverflow:
https://stackoverflow.com/questions/38394662/error-unknown-member-id-occurred-while-committing-offsets-for-group-xxx

Using the suggestion here I set the max_bytes to 500_000 and the lag that we had was consumed immediately. I wonder if this will last, but I'm really in the dark here :D .

from kaffe.

@CrWEE Interesting. We have a mix of settings for that, anywhere between 100k and 1M I think.

But are you saying that with setting max bytes, the consumers are no longer going through all the rebalacing reported above?

from kaffe.

OK, it's been an "off and on" thing for me as well.

from kaffe.

I've been looking into this one a bit and am able to occasionally reproduce the rebalance on 2.3.0 as well. It does look like the ruby-kafka gem is also having rebalancing issues as a result of a regression on 2.3.0. See zendesk/ruby-kafka#769 (comment) and https://issues.apache.org/jira/browse/KAFKA-8653.

As time permits, I'll continue to do some research into this but I think we'll know more when 2.3.1 is released.

from kaffe.

I'm running a 2.3.1 broker locally and a sample kaffe apps I built no longer repeatedly rebalances.

@CrWEE / @nemzsom - Could you see if the repeated rebalance issue still exists on Kafka 2.3.1?

from kaffe.

Thanks @nemzsom!

from kaffe.