[Bug]: in some conditions, `fluvio consumer -c <id> <topic>` does not record offsets about fluvio HOT 12 CLOSED

I think I would shorten the default flush interval, but a signal handler sounds good too.

A flush w/ signal handling sounds good too. ctrl+C is a little tricky because you often want to preserve the interrupt behavior at least on a second ctrl+c. Maybe a shutdown with flush on a Ctrl+D fits a graceful shutdown and flush a little better - and we leave ctrl+c alone?

We would preserve interruption with Ctrl+C of course. We can just give some time (1-2 seconds for example) to flush and after that period exit. And, the second Ctrl+C also can force exit immediately. So, single Ctrl+C - 1 second graceful period and exit, double Ctrl+C - exit immediately. Something, like this.

from fluvio.

Ok, I’m thinking the SPU should detect a disconnect from the consumer and flush. It’s probably the solution you are suggesting above.

Without handling this case, the feature feels broken.

The flush here is from the client to the SPU. Any detections on SPU side are irrelevant here.

from fluvio.

How the auto strategy works:

1. Commit happens when the next record is yelded.
2. Flush happens after a certain interval (10 seconds now by default).
3. Flush happens after a commit is triggered (see item 1 in this list).
4. Both commit and flush are called when the stream is finished.

So, when you use -d option, both commit and flush called in the end and it stores the last read offset regardless of all other logic.

Without -d it quickly read all records, the flush interval has not passed yet. Another record (after 10 seconds) would store the offset. On top of that, most probably, Ctrl+C was pressed to stop consuming. We don't have a signal handler, so it shuts down the process without giving a chance for the consumer to invoke item 4 logic in the list.

As a suggestion, we could add a handler for Ctrl+C signal to consume that will gently stop consuming allowing to flush offsets if needed. Wdyt @digikata

from fluvio.

Is the flush interval configurable?

from fluvio.

Is the flush interval configurable?

On CLI - no, only on API level.

from fluvio.

Ok, I’m thinking the SPU should detect a disconnect from the consumer and flush. It’s probably the solution you are suggesting above.

Without handling this case, the feature feels broken.

from fluvio.

I assume the implementation is acceptable for clients that have many knobs to manage flushing.

It's the CLI that feels odd. If I type a few records and wait for the flush to happen, it does not unless I send another record after 10 seconds.

I suggest changing CLI flushing to 1 or 2 second intervals. That would mask this behavior.'

Most users looking to test consumer offset at the CLI, won't send too many records, and we should optimize for that.

from fluvio.

I think I would shorten the default flush interval, but a signal handler sounds good too.

A flush w/ signal handling sounds good too. ctrl+C is a little tricky because you often want to preserve the interrupt behavior at least on a second ctrl+c. Maybe a shutdown with flush on a Ctrl+D fits a graceful shutdown and flush a little better - and we leave ctrl+c alone?

from fluvio.

disconnect or end of consumer session should trigger flusg

from fluvio.

disconnect or end of consumer session should trigger flusg

It does. The SIGINT signal does not leave much space for any code execution.

from fluvio.

Can we also change the flush interval to 2 sec ?

from fluvio.

Can we also change the flush interval to 2 sec ?

#3976

from fluvio.