Must have ability to checkpoint and continue on execution completion, by default about azure-functions-kafka-extension HOT 10 CLOSED

And to clarify the behavior of Event Hubs is:

1. Retrieve in batch (regardless of if single message function or batch message function)
   2a. If they have written their function to process individual messages

• Do a Task.WhenAll() and parallel-y invoke a function for each message in the batch. So if you got a batch of 30 messages, you would have 30 function invocations tasks kicked off and potentially processing in parallel.
  2b. If they have written their function to process a batch (recommended and default)
• Pass the entire batch into a single execution

3. Wait for batch to process (either all the concurrent executions or the single batch execution)
4. Checkpoint and retrieve the next batch

from azure-functions-kafka-extension.

Unlike EventHubs, Kafka provides checkpoint saving internally (saved as a compact topic). The idea is to use this but save it depending on how settings have been defined, right?

from azure-functions-kafka-extension.

I think this makes sense. The timing of the checkpoint is a bit important though. We want to have "at least once" delivery, so as long as the Kafka internal checkpointing allows us to "checkpoint" a batch after processing that should work great.

from azure-functions-kafka-extension.

Yes, we call commit topic/partition/offset. Current implementation does that after batch is consumed, but I need to add more customisation and retries.

from azure-functions-kafka-extension.

@fbeltrao does it checkpoint as soon as batch is read from Kafka, before it is processed?

from azure-functions-kafka-extension.

@fbeltrao let's chat about what "more customisation" we need in this first milestone. We can always come back and make it more customisable in future.

from azure-functions-kafka-extension.

My proposal:

• For triggers with batch (KafkaEventData[]) we commit after successfully triggering
• For single event triggers we commit every X times

from azure-functions-kafka-extension.

Batch sounds good

For single, I am not sure what you mean by every X times.
If I read 1 event from Kafka, process it do we not checkpoint then?
That means if another event later somewhere fails then this one we've already processed successfully will be replayed?

from azure-functions-kafka-extension.

Correct. Now that I think about, maybe is better to do the way you described, saving after each message, giving the most accurate solution out of the box.

Performance is bad, but anything less than trivial should be using batch anyway.

from azure-functions-kafka-extension.

Implemented as above

from azure-functions-kafka-extension.