Distributed ConsumerGroups about poseidon HOT 8 CLOSED

Hello, any idea if a similar gem exists for enabling the producer to auto-discover Kafka brokers via Zookeeper?

Thank you

from poseidon.

Not yet, but I would gladly merge a push request :)

from poseidon.

By the way @dim, I gave the poseidon_cluster gem a try.

Unfortunately, I ended up removing it because it depends on zk and zk currently has a deprecated dependency on an older logging gem version (zk-ruby/zk#79). I was also unable to actually fetch anything from my kafka nodes, but this is probably an error on my side.

Is there an issue queue for poseidon_cluster?

Finally, is there a reason why it requires both the zookeeper and kafka hosts?
Shouldn't it be either the one or the other?

Thank you

from poseidon.

Can I infer from this discussion that Poseidon does not support consumer groups? I looked through the code and didn't see anything that would lead me to believe it does, but just wanted to be sure.

from poseidon.

Hi,

I tried to keep poseidon cluster as compatible with Kafka's own consumer
implementation as possible, following guidelines outlined in
http://kafka.apache.org/documentation.html#distributionimpl. Hence, zk
dependency is a necessity.

I also added issue queue to the project.
On 10 Mar 2014 11:07, "yourtallness" [email protected] wrote:

By the way @dim https://github.com/dim, I gave the poseidon_cluster gem
a try.

Unfortunately, I ended up removing it because it depends on zk and zk
currently has a deprecated dependency on an older logging gem version (
zk-ruby/zk#79 zk-ruby/zk#79). I was also
unable to actually fetch anything from my kafka nodes, but this is probably
an error on my side.

Is there an issue queue for poseidon_cluster?

Finally, is there a reason why it requires both the zookeeper and kafka
hosts?
Shouldn't it be either the one or the other?

Thank you

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/29#issuecomment-37171593
.

from poseidon.

@yourtallness zk-ruby/zk#79 is closed, use zk-1.9.4

from poseidon.

@dim is your plan to keep this a separate project or submit a PR to posidon?

from poseidon.

The reason why I wanted to keep it separate is the mentioned zk dependency. The consumer group/clustering functionality is also quite specific and may not be relevant for the average user.

from poseidon.