Must have an entire batch processed before the next batch can be pulled about azure-functions-kafka-extension HOT 5 CLOSED

@jeffhollan So we hold a lock on the partition / stream, until a batch is complete, then release that lock?
What if the batch fails and things go pear shaped, how do we release that lock and let another instance start a new lock?

from azure-functions-kafka-extension.

We should wait for current batch on instance to FULLY complete before moving to the next. That’s the only way we can guarantee ordering which is often a big deal for these stream scenarios.

To the point of if a batch goes south and hangs, that’s a fair point. In the event hubs trigger I BELIEVE a function timeout will result in a checkpoint and continue. So in a non-consumption world that defaults to 30 min we’d cut the execution.

Honestly I’m not sure there is a great way to deal with a bad batch other than hoping people write safe code. Even if we kept dequeuing batches it’s possible we’d just be piling up executions if I had a bug in my code and all batches loop infinitely. So for simplicity hoping we mirror event hubs behavior which is 1 batch at a time (though event hubs does give you the option to process each message in the batch concurrently or as a whole — as mentioned the “as a whole” batch process should be the P0 and default)

from azure-functions-kafka-extension.

I foresee us doing a COMMIT OFFSET then acknowledge on consumer thread. There are concerns with race conditions especially around multiple partitions, so we might need to think of a good solution to solve it.

While it's important to point out that Kafka guarantees item ordering in the order it was written, not necessarily in the intended order. We can "guarantee" ordering by providing a key and the producer will use that key in the default partition algorithm.

A deterministic algorithm as such "hash(key)%numPartitions = partition" - if the key doesn't change and the partition count doesn't change the records will always be written to the same partition, and Kafka will store the records in the order it receives them.

Good practice is to ensure we have retry policies with Dead Letter Queue to handle consumer failure, and Circuit Breaker pattern to handle producer issues.

There are caveats to this, while we maintain order, parallelism of tasks is compromised.

from azure-functions-kafka-extension.

@kennetham how does this compare with the current Event Hubs trigger? For P0 we're trying to replicate that behavior, where it makes sense, for Kafka. We can always come back and add additional capabilities etc. as future improvements.

@jeffhollan thoughts?

from azure-functions-kafka-extension.

Most of what @kennetham is true for Event Hubs as well, the ordering guarantees being on the order received which is generally adequate especially when you have multiple partition keys that hash to the same partition.

I'm not sure about the COMMIT stuff. For Event Hubs only one consumer is allowed on a single partition at a time, so you don't really have to worry about race conditions. I understood Kafka was the same. So while you may have many parallel instances consuming from the same stream / consumer group, only one instance is active on one partition at a time.

I should clarify a batch would only be a batch for a partition, and we should only expect order to be guaranteed per partition. This maps to Event Hubs as well.

from azure-functions-kafka-extension.