Middleware about racecar HOT 9 CLOSED

Yup. Issue #61 describes how user code could trigger a heartbeat, but that's not implemented yet.

from racecar.

The problem here that I see here is with how messages are being handled from Kafka itself. You can either process single messages or do that per batches. You case is valid for a single message but not for batches. Furthermore with the incoming ruby-kafka 0.7 there will be new mesasges format, batch headers involved and couple other things. So the real question would be, where to put the middleware to make in functional and generic. TBH I would be in favor of using events / instrumentation approach to build up loggers and other things instead of injecting that somewhere in the framework itself. Not to mention that yield is not really too fast when invoked a lot.

from racecar.

Thanks for your comment, I'm going to try and cover all your points here:

You can either process single messages or do that per batches.

Good point! The middleware required for a consumer might need to be different for single message vs batch message.

Furthermore with the incoming ruby-kafka 0.7 there will be new mesasges format, batch headers involved and couple other things.

Regarding the new format, I'm not sure how that changes things with middleware - can you explain further?

So the real question would be, where to put the middleware to make in functional and generic.

Yes, this is a design decision that I don't know just yet. Without knowing the racecar internals, I'd guess we would apply the middleware stack once the message struct is instantiated. I will dive into the exact implementation details once I know the team is interested in it.

TBH I would be in favor of using events / instrumentation approach to build up loggers and other things instead of injecting that somewhere in the framework itself.

I wouldn't read into my example as the exact/best use case - it was just a simple example. Middleware has many benefits, especially for common error handling, set ups, tear downs and monitoring within the consumption or producing lifecycle.

Not to mention that yield is not really too fast when invoked a lot.

Are you suggesting the middleware proposal should avoid blocks?

from racecar.

Good point! The middleware required for a consumer might need to be different for single message vs batch message.

And this already makes things more complex. Racecar was meant to be a simple framework for Kafka consumers in Ruby. Middlewares are great when the flow is linear and here, due to different strategies you can apply (each_message, each_batch, etc) you would have to distinguish between them and this becomes complex.

Regarding the new format, I'm not sure how that changes things with middleware - can you explain further?

batch headers. This goes back to having a different API with different args for each case. More cases, more middleware versions to handle.

Yes, this is a design decision that I don't know just yet. Without knowing the racecar internals, I'd guess we would apply the middleware stack once the message struct is instantiated.

This happens in ruby-kafka together with the headers and some metadata available only for batch fetches.

I wouldn't read into my example as the exact/best use case

Could you provide the best use case then? It is really hard to discuss something based on examples to which even the author refers as not the best ;)

Middleware has many benefits, especially for common error handling, set ups, tear downs and monitoring.

Not really. Middlewares are meant to provide you with an option to change/impact the system flow not to provide some additional logging, instrumentation or error handling. For all those cases (apart from error handling) you can use instrumentation and hook up to proper events. Ruby-kafka error handling is pretty good, so you should focus more on your business logic so it won't leak (if not needed) to racecar. Using middleware in the way you describe, without actually changing the framework behavior for me is an antipattern as it hides some "semi-app" logic within the framework core. Your business should not leak to the framework (nor the other way around).

Are you suggesting the middleware proposal should avoid blocks?

I'm pointing out, that yield has a certain price that should be justified.

from racecar.

Can you come up with some examples that better represent a value-add over e.g. async event subscribers (which should be possible already using Active Support Notifications)?

from racecar.

Yes I'm happy to provide some additional context, I have two real world examples that we would use for certain consumers.

1. A lock around a shared resource:

class MyLockMiddleware
  def call(consumer, message)
    Redis::Lock.new(redis_client).lock do |lock|
      yield
    end
  end
end

2. A [distributed] circuit breaker that each job runs in:

class MyBreakerMiddleware
  def call(consumer, message, &block)
    my_circuit_breaker = block
    my_circuit_breaker.call(consumer, message)
  end
end

I think the second example I could rework to use ActiveSupport::Notifications but I would still need to prepend a module to check the breaker status before running the consumer.

from racecar.

What would the circuit breaker middleware do if the circuit opens? Raise an exception? Sleep?

from racecar.

So for the locking middleware – inside process you can't just pause, as that would kick the consumer out of the group.

from racecar.

Circuit breaker - if the lock duration is usually small we will rescue the open exception and sleep.

Locking - Ah interesting. What triggers the kick? Lack of a heartbeat?

from racecar.