[Question] Multiple consumers multiple Kafka clusters about kaffe HOT 6 CLOSED

Hi @sweco-semtne,

Unfortunately it isn't possible at the moment. If @sdball / @objectuser aren't opposed to it, I'd be happy to add an additional start_link to Kaffe.Consumer that allowed a configuration to be passed in, rather than using Kaffe.Consumer.Config

This is a use-case I have too, by the way. For the time being, I've adding my extra consumers directly through :brod

from kaffe.

So the only change would be the addition of a start_link(config) function that would call the same start_link_group_subscriber function with the salient pieces? That sounds entirely reasonable to me.

Then you could have several worker declarations, each its own consumer and specialized config. That's a great impact for a minimal code change! 👍

from kaffe.

@sweco-semtne If you're using Kaffe.Consumer, the solution proposed by @rawkode does sound like an easy win.

I wanted to note that there are two modes of consuming messages in Kaffe and the Kaffe.GroupMemberSupervisor has some advantages discussed here. However, it wouldn't be quite as straight forward to allow it to consume from multiple clusters.

from kaffe.

Thanks!
Kaffe.GroupMemberSupervisor seems interesting.
How would you setup multiple Kaffe.GroupMemberSupervisor with multiple topics (using the same cluster)?

from kaffe.

@sweco-semtne The topics list in the consumer config should allow for multiple topics, using only a single GroupMemberSupervisor. It's only if you're trying to use two clusters in the node that Kaffe would need an enhancement:

config :kaffe,
  consumer: [
    heroku_kafka_env: true,
    topics: ["topic1", "topic2"],
    consumer_group: "my-group",
    message_handler: MyHandlerModule,
  ]

Is that answering your question? If you really need two GroupMemberSupervisors, it's a bit tricker, but GroupMemberStartupTest might provide a work-around.

from kaffe.

Closing stale question.

from kaffe.