Update to brod 2.4.1 about kaffe HOT 7 CLOSED

Oh I think you're right! Ah yes I even saw that behavior locally when I testing buffering vs not. When you produce sync from separate processes brod can allow things to buffer internally or not.

from kaffe.

@objectuser It looks like brod has some new settings based around controlling brod's internal buffering of messages. Maybe they could be used to make our message production more efficient.

%%   max_linger_ms(optional, default = 0):
%%     Messages are allowed to 'linger' in buffer for this amount of
%%     milli-seconds before being sent.
%%     Definition of 'linger': A message is in 'linger' state when it is allowed
%%     to be sent on-wire, but chosen not to (for better batching).
%%     The default value is 0 for 2 reasons:
%%     1. Backward compatibility (for 2.x releases)
%%     2. Not to surprise `brod:produce_sync' callers
%%   max_linger_count(optional, default = 0):
%%     At most this amount (count not size) of messages are allowed to 'linger'
%%     in buffer. Messages will be sent regardless of 'linger' age when this
%%     threshold is hit.
%%     NOTE: It does not make sense to have this value set larger than
%%           `partition_buffer_limit'

from kaffe.

@sdball I had looked at those previously. If I understand them correctly, it's like implicit buffering. If we used them, we could potentially produce_sync, then ack an offset, crash and not write the messages.

Is that your understanding?

from kaffe.

@objectuser Yes that's possible, depending on how we control the ACKs back to Kafka. We'd want to ensure we only ACK the highest offset that we can definitively place at the end of a contiguous block of offsets.

Or write missed offsets to another system to be individually fetched and replayed?

from kaffe.

To be clear, these new fields only come into play if we use :brod.produce_async

from kaffe.

@sdball Based on the quoted reasoning I was thinking they could also be used with :brod.produce_sync:

2. Not to surprise brod:produce_sync callers

But maybe I'm misinterpreting.

Still, I think brod:produce_sync with some batching is a good next step!

from kaffe.

Kaffe currently supports the latest Brod.

from kaffe.