Consumer registration and rebalancing about sarama HOT 11 CLOSED

I went ahead and created a library on top of sarama: http://godoc.org/github.com/bsm/sarama/cluster

It integrates Sarama with Zookeeper and implements Consumer Group functionality. Should it be merged into the main sarama tree or do you want me to maintain it separately?

from sarama.

Sorry for the very late reply. I think we should include this in Sarama: without it, you don't get many of the advantages that Kafka offers. Let's get it merged!

from sarama.

So I'm still working out the API. While the code works just fine, my main concern is currently error/event handling. There is no concise way to feed back errors and state changes to the client. I could sneak in a few ConsumerEvents with an Err, but I would almost prefer to allow users to register event listener callbacks. Thoughts?

from sarama.

Maybe we can do something like this:

cg.Process(chan sarama.ConsumerEvent, chan error)

from sarama.

very nice but the name collision makes things a little annoying to pull a fork.

On Mar 19, 2014, at 11:17 AM, Dimitrij Denissenko [email protected] wrote:

I went ahead and created a library on top of sarama: http://godoc.org/github.com/bsm/sarama/cluster

It integrates Sarama with Zookeeper and implements Consumer Group functionality.

—
Reply to this email directly or view it on GitHub.

from sarama.

I will start the proces sof moving this code over in a branch.

from sarama.

See #84

from sarama.

Thanks. I also added a more generic event notification in bsm/sarama@3ad706907810c16.

from sarama.

I think it's better to keep this separate from the main sarama code and not pollute it with zk dependencies. I have created a new project for it: https://github.com/bsm/sarama-cluster

from sarama.

@dim now that zookeeper isn't a dependency to implement consumer groups, what are the chances you can move this into the official sarama repo?

from sarama.

@arianitu maybe, but there is quite a bit of effort involved and the integration is not 100% straight-forwards as the test cases/setups are more complicated. more importantly: I probably won't be able to find the time for such an effort any time soon, but help is more than welcome

from sarama.