[Question] dynamically registering new topics about kaffe HOT 6 OPEN

Kaffe doesn't support dynamic configuration. However, there was some discussions around using a library called Weave for that. I've not used it myself, but we did have a PR on it some time ago.

from kaffe.

thanks. I'm going to hack kaffe to make it dynamic, as I have a pool of boxes that I need to consume various topics at different time. I'm going to use Swarm for the distributed aspect of it. I think it'll be easy-ish to make kaffe dynamic. I'll let you know here. Feel free to close

from kaffe.

see #58

from kaffe.

@dams thanks for the pings in the dynamic topic subscribe/unsubscribe PRs. Sorry for the delay, we've been busy here at Spreedly the last few months and are just starting to look back at continuing development on Kaffe.

We were chatting about this internally and thought we'd get a little more details. We're curious about use cases in which systems are only occasionally consuming a topic. Could be that this is a domain specific situation, of if there are other factors that make this a more common scenario and we're just not experiencing it within our system.

I do think it's worth nothing that this particular behavior could also be managed on the message handler side of things by ignoring messages from a particular topic if certain criteria are true (such as the time of day, etc). Of course the criteria would need to be static and maybe your particular case won't allow it to be handled in your message handler code?

from kaffe.

It’s not a domain specific situation. To me it is mandatory that à Kafka client allows this. The official clients in other languages do provide this.

My need is very common and simple: provide a feature ( here an aggregation over time ) of data transported over Kafka, * as a service *. In the current world, things are service oriented, as a service, with zero downtime.

My solution will run 24/7 with hot code reloading. It’ll run for the next years without being restarted. So it needs to be able to adapt to the change in flow if data, namely new topics, new partitions, etc.

Your workaround works only if all the topics exist at start time, which is not the case.

This feature is mandatory imho to be able to provide a strong kafka client.

Also, other PRs by other authors are important too, like the one that allows to have more than one consumer group

from kaffe.

That makes sense. I think since we're operating slightly differently at the moment so it wasn't something that had quite caught our attention. But I can imagine sometime in the future we'd make use of dynamic topics. Just not right now. I'll give your PR another look next week and see if any changes need to happen before merging.

Thanks!

from kaffe.