Comments (3)
I've opened #93 to add those functions specifically, and #94 to track remaining functions.
from fs2-kafka.
Thanks for raising this issue @backuitist!
Currently the
KafkaConsumer
exposes only a tiny subset of all the apache kafka consumer methods.
That's correct, and it's mainly because no-one has asked for or implemented the remaining functions. It would be helpful if you could either (1) list the non-implemented functions you need, or (2) list all the non-implemented functions for reference, and we could track them here.
Wouldn't it be easier to provide an (albeit dangerous)
withConsumer
(similar to the one onKafkaConsumerActor
) instead of actor style communication?
One of the design decisions is to not expose impure functions, which you would do indirectly by providing a withConsumer
function. My preference is that we expose pure versions of those functions directly on KafkaConsumer
instead.
Speaking of which, what's the purpose of the actor since you anyway
synchronize
calls to the underlying apache kafka consumer?
The initial implementation started out with the 'actor' approach, but then the synchronized construct was added to deal with finalizer behaviour (calling close
on the Java Kafka consumer). You're right in that we can get rid of the actor approach, relying solely on Synchronized
, but it simply has not been done yet. However, there needs to be some communication between the poll-loop and some of the requests (e.g. subscribe and fetch).
from fs2-kafka.
Thanks for the quick and thorough answer!
Right now my use-case is to read the last x
events of a topic. Here are the methods of interest:
Set<TopicPartition> assignment();
void seekToEnd(Collection<TopicPartition> partitions);
-- that one is easily implemented withendOffsets
+seek
Potentially useful:
long position(TopicPartition partition);
from fs2-kafka.
Related Issues (20)
- Records are missing when second rebalance happens. HOT 6
- Is there a way to check if the producer is ready to produce?
- One message from Consumer? HOT 6
- withCreateConsumer and withCreateProducter methods have been removed HOT 13
- TransactionalKafkaProducer don't commit offset with empty records HOT 2
- Does KafkaConsumer.stopConsuming actually allow to process all in-flight messages? HOT 7
- It would be nice to have a `.option` method on `RecordSerializer` HOT 2
- Using Codec[A] with KafkaStreams
- CE 3.5.0 async cancelation semantics HOT 3
- Question around custom Partition assignments HOT 3
- Multiple consumers read from the same partition after rebalance HOT 2
- Any reason `RebalanceInProgressException` isn't considered in`CommitRecovery.Default`? HOT 1
- Confluent platform incompatibility HOT 3
- Stuck partitions - question about the pause/resume logic HOT 20
- upgrade 3.x to CE 3.5.1 / fs2 3.8.0 HOT 5
- Feature request (possibly) : Dynamic metering on Streams HOT 2
- retry CommitTimeoutException
- KafkaConsumer closed prematurely on finite streams
- Zipping finite KafkaConsumer-based streams doesn't behave as expected (first stream gets recycled after first 1000 elements) HOT 1
- Performance regression Producer 3.5.0 vs 3.4.0 HOT 9
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fs2-kafka.