Missing documentation about kaffe HOT 8 CLOSED

@fatcatt316 I haven't gone back to this side project to see if it works. We can close it for now and I'll reopen if I have more questions. Thanks.

from kaffe.

@batasrki Doesn't this portion cover that?

:ok - commit back the most recent offset and request more messages {:ok, :no_commit} - do not commit back the most recent offset and request more message from the offset of the last message {:ok, offset} - commit back at the offset specified and request messages from that point forward

Basically, if you're ready to commit, return :ok or {:ok, offset} instead.

from kaffe.

Hey @objectuser,

Sorry, I didn't give the full context. I would like to build a webhook delivery service where webhooks are sent to my service through Kafka. In order to have this service fairly scalable, the design I'm thinking of is:

1. Receive a batch of messages from Kafka
2. For each message, spawn a GenServer and give it the message
3. GenServer attempts HTTP delivery, forwards on the result

In this case, each message isn't done processing until step 3. However, since we're talking about HTTP delivery, a request can take up to some amount of time (I'm capping mine at 5 seconds) to complete. With a Kafka-based system, that means that there'll be head-of-line blocking for up to 5 seconds.

To compensate for that, the design calls for async offset committing. Each GenServer would need to notify some other component that its specific offset has done processing, and that other component then needs to notify Kafka.

The first half of that is using the {:ok, :no_commit} response. The first paragraph of that section is essentially talking about my use case. This is why I asked the question around what does one do in that case. A piece of code somewhere has to notify something in Kaffe that an offset can be committed. Otherwise, there's topic lag up to the retention period.

I thought that Kaffe.Subscriber.commit_offsets/5 or Kaffe.Consumer.ack/2 would work, but within the message handler module, I don't have an ability to reference either of these. Also, using Kaffe.Consumer is deprecated as per README.

from kaffe.

@batasrki I'm not sure I fully understand but it sounds like you'd like to commit offsets in any order.

If true, I think that's counter to how Kafka works. Committing any specific offset implies committing all prior offsets.

from kaffe.

I'm not sure I fully understand but it sounds like you'd like to commit offsets in any order.

Because messages are fanned out across many GenServers, offsets to be committed will come back in any order. However, my service will commit the highest contiguous offset it can. This has to be done on a timer or something, but it's basically an implementation detail. It implies tracking what offsets are ready to be committed and finding the highest appropriate one from that set.

The question is what mechanism within Kaffe can I use to commit an offset, if I use {:ok, :no_commit} as the response from my message handler?

from kaffe.

Ah, maybe I understand. You'd like to send a message somewhere to say, "OK, go ahead and commit up to this offset", separately from the Kafka consumer's message handler function. Is that right?

Currently the Kaffe answer is to return {:ok, offset}, which doesn't sound ideal in your circumstance.

If your message handler is a GenServer instance, however, maybe you could send messages to that and the next time it gets a message batch from Kafka, it could commit the required offsets.

I'm not sure I can think of another way. Maybe exploring the processes would suggest an approach (with some updates to the code, perhaps).

from kaffe.

You'd like to send a message somewhere to say, "OK, go ahead and commit up to this offset", separately from the Kafka consumer's message handler function.

Yep, that's correct!

If your message handler is a GenServer instance, however, maybe you could send messages to that and the next time it gets a message batch from Kafka, it could commit the required offsets.

Interesting! That would work, I think. Totally worth a shot.

from kaffe.

@batasrki Did that work for you? If so, I'm wondering if this issue could be closed.

from kaffe.