Missing messages in consumer about poseidon HOT 5 OPEN

Do you have multiple Kafka brokers/partitions? Could be you're just consuming messages from a single partition.

from poseidon.

@damien Thanks for the response.

I did not set a partitioner for the producer so it seems that it is publishing messages to more than one partition. Can I have a consumer to consume all messages from a topic regardless of the partition? Or at least a way to know how many partitions are there for a topic.

Thanks

from poseidon.

Unfortunately, consuming messages from Kafka is quite a bit more complicated than publishing messages to Kafka. I'm in the process of figuring out a similar problem myself and the best ruby based consumer library I've been able to find for the problem is poseidon_cluster.

Setting that up and pointing it at your brokers should ensure you get all the messages you're looking for; otherwise you probably really are seeing a bug in poseidon itself.

from poseidon.

I did a workaround which I think is wrong in general. I set the partitioner so that all messages are published to one partition. Then I create a partition consumer for that partition.

from poseidon.

That does seem to verify that you were only listening to one of multiple partitions. Definitely check out poseidon_cluster; your workaround may fix your problem but you'll be giving up a lot of Kafka's guarantees around persistence/availability if you're running off of a single partition.

from poseidon.