Implement an asynchronous producer about ruby-kafka HOT 10 CLOSED

Do you know how do kafka-clients of other language such as Java, Python implement this function?

from ruby-kafka.

The Java client has a background thread, I believe. However, I'd like to make the Ruby version more composable.

from ruby-kafka.

@dasch you can take a look into celluloid - it might help you out to provide an abstraction layer for that

from ruby-kafka.

@mensfeld I'd like to avoid adding heavy dependencies (such as Celluloid) to this library. I think I'm going to punt on this for a while.

from ruby-kafka.

@dasch you could also do what newrelic is doing for their monitoring stuff - they spin up a thread and push stuff there that gets flushed to their servers from time to time

from ruby-kafka.

Yeah, that's my plan. I think I'll add something to our web app first, see how it fares, and then maybe extract it into ruby-kafka. My main concerns are getting the periodic wakeup to work and ensuring that messages are sent on shutdown.

from ruby-kafka.

ensuring that messages are sent on shutdown.

You could just expose a trigger for that action and the shutdown implementation could differ for apps that use it. Something like (naming taken from newrelic rpm): Kafka.harvester.flush!

from ruby-kafka.

We have a similar abstraction in our code base: KafkaLog. It's a singleton and you call flush on it.

from ruby-kafka.

@dasch just remember to add a mutex (probably don't have to tell you that but oh well ;) ) - so a multiple flash from many threads won't occur

from ruby-kafka.

Yeah, the flush would happen from the background thread itself, so that wouldn't be an issue. I think I'll just add a thread-safe queue between the main and the background threads, that usually makes things simpler.

from ruby-kafka.