kambing86 / kafkajs-stream Goto Github PK

Hi! I've implemented a consumer stream based on your code and found that skipping batch messages can lead to huge performance issues in the case of a large batch. For example, if the stream client limits the load by pausing/resuming the stream every 10 messages and the batch contains 1,000 messages, then the Kafka client will load 1,000, 990, 980, and so on. To fix this, you should try to wait for the already loaded batch to resume instead of skipping it (but not for longer than the heartbeat interval).

Hi,

thanks great for the work. It's a perfect lib for using kafkajs together with stream. However, could you provide more options to set autoCommit relevant flags. Just like the configs of kafkajs:

Manual committing

consumer.run({
    autoCommit: false,
    eachMessage: async ({ topic, partition, message }) => {
        // Process the message somehow
    },
})

We want to consume the kafka without offset commit. If possible, sincerely thousand thanks.

Best regards,
Kevin