consumeStrings() discards buffered messages about kafka-junit HOT 3 OPEN

consumeStrings is a convenience method that makes a lot of assumptions so it probably won't suit some use cases. You can create your own consumer by calling createStringConsumer or createConsumer with the desired configuration and use that instead.

from kafka-junit.

But that method is practically unusable when trying to read less message than there are in topic. The rest will be always lost. And if that is desired behavior, it should be noted in javadoc since I spent more than hour trying to find out what's going on. The same applies to consume() method itself. When I pass consumer with autocommit into it, it will discard some messages. Also should have some note in documentation.

from kafka-junit.

It's not desired behaviour but rather an unfortunate side effect. The consume* helpers are something I regret in hindsight so it's probably time to deprecate them. They make a lot of assumptions about their usage and haven't aged well.

I am not sure it's possible to document all possible ways in which they could go wrong but I'd be happy to accept a PR if you have any suggestions.

I'd suggest using the following pattern as an immediate workaround for your problem:

KafkaConsumer<String, String> consumer = kafkaRule.helper().createStringConsumer(kafkaRule.helper().consumerConfig(false));
List<String> result = kafkaRule.helper().consume("my-test-topic", consumer,3).get().stream().map(ConsumerRecord::value).collect(Collectors.toList());

from kafka-junit.