Expose more of the underlying apache consumer methods about fs2-kafka HOT 3 CLOSED

I've opened #93 to add those functions specifically, and #94 to track remaining functions.

from fs2-kafka.

Thanks for raising this issue @backuitist!

Currently the KafkaConsumer exposes only a tiny subset of all the apache kafka consumer methods.

That's correct, and it's mainly because no-one has asked for or implemented the remaining functions. It would be helpful if you could either (1) list the non-implemented functions you need, or (2) list all the non-implemented functions for reference, and we could track them here.

Wouldn't it be easier to provide an (albeit dangerous) withConsumer (similar to the one on KafkaConsumerActor) instead of actor style communication?

One of the design decisions is to not expose impure functions, which you would do indirectly by providing a withConsumer function. My preference is that we expose pure versions of those functions directly on KafkaConsumer instead.

Speaking of which, what's the purpose of the actor since you anyway synchronize calls to the underlying apache kafka consumer?

The initial implementation started out with the 'actor' approach, but then the synchronized construct was added to deal with finalizer behaviour (calling close on the Java Kafka consumer). You're right in that we can get rid of the actor approach, relying solely on Synchronized, but it simply has not been done yet. However, there needs to be some communication between the poll-loop and some of the requests (e.g. subscribe and fetch).

from fs2-kafka.

Thanks for the quick and thorough answer!
Right now my use-case is to read the last x events of a topic. Here are the methods of interest:

• Set<TopicPartition> assignment();
• void seekToEnd(Collection<TopicPartition> partitions); -- that one is easily implemented with endOffsets + seek

Potentially useful:

• long position(TopicPartition partition);

from fs2-kafka.