Comments (11)
I went ahead and created a library on top of sarama: http://godoc.org/github.com/bsm/sarama/cluster
It integrates Sarama with Zookeeper and implements Consumer Group functionality. Should it be merged into the main sarama tree or do you want me to maintain it separately?
from sarama.
Sorry for the very late reply. I think we should include this in Sarama: without it, you don't get many of the advantages that Kafka offers. Let's get it merged!
from sarama.
So I'm still working out the API. While the code works just fine, my main concern is currently error/event handling. There is no concise way to feed back errors and state changes to the client. I could sneak in a few ConsumerEvents with an Err, but I would almost prefer to allow users to register event listener callbacks. Thoughts?
from sarama.
Maybe we can do something like this:
cg.Process(chan sarama.ConsumerEvent, chan error)
from sarama.
very nice but the name collision makes things a little annoying to pull a fork.
On Mar 19, 2014, at 11:17 AM, Dimitrij Denissenko [email protected] wrote:
I went ahead and created a library on top of sarama: http://godoc.org/github.com/bsm/sarama/cluster
It integrates Sarama with Zookeeper and implements Consumer Group functionality.
—
Reply to this email directly or view it on GitHub.
from sarama.
I will start the proces sof moving this code over in a branch.
from sarama.
See #84
from sarama.
Thanks. I also added a more generic event notification in bsm/sarama@3ad706907810c16.
from sarama.
I think it's better to keep this separate from the main sarama code and not pollute it with zk dependencies. I have created a new project for it: https://github.com/bsm/sarama-cluster
from sarama.
@dim now that zookeeper isn't a dependency to implement consumer groups, what are the chances you can move this into the official sarama repo?
from sarama.
@arianitu maybe, but there is quite a bit of effort involved and the integration is not 100% straight-forwards as the test cases/setups are more complicated. more importantly: I probably won't be able to find the time for such an effort any time soon, but help is more than welcome
from sarama.
Related Issues (20)
- Bug while adding new ACLs HOT 5
- sarama.CreateAclsRequest: Ignoring TextMarshaler & TextUnmarshaler for members HOT 2
- Kafka: client has run out of available brokers HOT 3
- Should share the max retries of metadata? HOT 2
- Topic retention settings causes consumer to skip consuming new messages that arrived after deleted messages HOT 5
- A topic with multiple partitions won't get dedicated consumer members in consumer group for each partition HOT 6
- kafka: Failed to produce message to topic myRsp: write tcp [ip1]:[port1]->[ip2]:[port2]: write: broken pipe HOT 7
- Enabling idempotency still allows producing same message
- No throughput difference between sync and async producers HOT 1
- High CPU usage in `newBrokerProducer.func2` HOT 3
- Incorrect rate5 metrics values
- Error and data race using transaction example from the library HOT 2
- kafka server: The producer attempted to update a transaction while another concurrent operation on the same transaction was ongoing HOT 3
- Will handling response this way result in subsequent requests all being mishandled? HOT 1
- unsubscribe w/o passing whole list HOT 2
- Sincerely ask, is this redundant? HOT 1
- How can consumer group consume new partition without losing data using OffsetNewest config?
- Setting a too high value for Producer.Retry.Max can lead to excessive memory usage.
- Docs with examples using tls/ssl by cert files creating a producer and consumer HOT 1
- Broken pipe and EOF
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sarama.