[Bug]: KafkaRebalance resource reconciliation gets stuck if CC pod is restarted during rebalance about strimzi-kafka-operator HOT 10 OPEN

Thanks, fixed the formatting.

from strimzi-kafka-operator.

Thanks for raising the issue. Could you please format the YAMLs to make them readable? Thanks.

from strimzi-kafka-operator.

@urbandan I tried it with the latest operator(0.41.0) and was not able to get this error. For me when you increase the node pool brokers to 4 then the KafkaRebalance moves into NotReady state since the Kafka cluster is in the NotReady state as the pods are still coming which should be what we desire. Then once the Kafka cluster is up with all pods running then the KafkaRebalance will move again to ProposalReady state. I will try with 0.40.0 now

from strimzi-kafka-operator.

@ShubhamRwt Why would the Kafka cluster move to NotReady just because you scaled up the node pool to 4 nodes? That sounds like you had some other issue interfering with the reproducer.

from strimzi-kafka-operator.

@scholzj I meant when we scale up the nodes then pods corresponding to cruise-control and the 4th node will be coming up. It would mean that the kafka cluster is not ready yet so there we have the logic in the Rebalance operator that if the Kafka cluster is not up yet then we say Kafka cluster is not Ready.

from strimzi-kafka-operator.

No, the Kafka cluster should stay in Ready while scaling up.

from strimzi-kafka-operator.

Also, look at it differently -> what happens if you simply delete the CC pod while a rebalance is in progress?

from strimzi-kafka-operator.

Discussed on the community call on 16.5.2024: (Assuming this can be reproduced - see the discussion above), this should be addressed by failing the rebalance or restarting the process if possible. (Let's keep it in triage for next time to make sure it is reproducible and discuss the options)

Note: This should be already handled by the Topic Operator when changing the replication factor, where the TO detects this and automatically restarts the RF change. @fvaleri will double-check.

from strimzi-kafka-operator.

Yeah, we have this corner case covered in the Topic Operator.

TL/DR: Cruise Control has no memory of the task that it was working on before restart, so the operator is responsible for detecting this event and resubmit any ongoing task.

The operator periodically calls the user_tasks endpoint with one or more Cruise Control generated User-Task-ID, in order to check the status of pending tasks. If it gets back an empty task list, then it means that Cruise Control has restarted, and the tasks may or may not have been completed. This means that the operator must reset its internal state (switch back the resource state from ongoing to pending), and resubmit the tasks.

Note that there is a small chance that the task could have been completed just before Cruise Control restarted, but the operator didn't had time to know that. In this case, the new task submission would be a duplicate. This is not a problem in practice, as the work has already been done, and the duplicated task would be completed quickly (no-op).

from strimzi-kafka-operator.