Parallel processing of certain topics about racecar HOT 1 CLOSED

There are two basic approaches to getting more parallelism out of Kafka consumers:

1. Add more partitions and group members. In theory, you can have as many consumers as you have partitions, and each will run in parallel.
2. Use multiple threads in your consumers.

The second option is a bit dangerous, as you lose a bunch of guarantees that Kafka makes, such as ordering, but if you're okay with possibly reprocessing the same messages quite a few times when there's an error, this is a pretty simple approach:

class ParallelConsumer < Racecar::Consumer
  subscribes_to "work"

  def process_batch(batch)
    threads = batch.messages.map {|message|
      Thread.new { work(message) }
    }

    # Wait for all threads to finish before proceeding.
    threads.each(&:join)

    # Racecar will only checkpoint the consumer position when we return,
    # so if one of the threads fail, all will be retried.
  end
end

Note that this introduces concurrency, so you want to be careful and not share mutable state between the threads.

from racecar.