Exception: Unsupported compression type about racecar HOT 10 CLOSED

Good one, we set it to lz4 everywhere. I'll have a look

from racecar.

I wonder how compression comes into play on the consumer side. According to https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md the compression parameters can only be set for producers.

from racecar.

🤷‍♂

I'm not even sure why this happened – it was during broker shutdown. Could be an incorrect error message.

from racecar.

I’m on mobile right now, but generally speaking printing the error code might be helpful. Depending on the versions in use there might be a disagreement on what an error code means, see e.g. confluentinc/librdkafka#2245

from racecar.

I don't know how to reproduce this.

Using Kafka 2.3.0 / Zookeeper 3.4.13 from https://github.com/wurstmeister/kafka-docker in single broker mode (docker-compose-single-broker.yml with KAFKA_ADVERTISED_HOST_NAME: 127.0.0.1) I manage to get a

Rdkafka::RdkafkaError: Local: Broker transport failure (transport)

with essentially the same stack trace (as in the initial comment) when

1. docker-compose -f docker-compose-single-broker.yml up
2. start racecar consumer
3. CTRL+C docker compose

This doesn't always work for some reason (timing?), but the error message seems decent enough. The transport failure one is also years old, so it's unlikely I'm hitting a code path with changed/unreleased/mismatched error codes from Kafka and librdkafka.

Can you provide more details about your local setup? Generally speaking I'm hesitant to add re-connect in these scenarios because resetting librdkafka's internal offset state is not trivial.

from racecar.

I can try to see if I can hit the condition again.

However – the consumers are supposed to be resilient to temporary cluster failures. I think we need some kind of automatic retries.

from racecar.

I think that let it crash is a reasonable strategy for unknown and unpredictable errors. It is advisable anyway to have a supervising process that does restarts (with exponential back-off) and alerting.

When there is a possibility to reproduce the condition, then it makes sense to handle it here. Probably catch/retry this specific exception + error code around poll in the ConsumerSet. Otherwise the situation could get worse, as @breunigs described with unknowns in the internal state of rdkafka.

I think that we should not introduce a generic mechanism for automatic retries.

from racecar.

Is Broker transport failure not a temporary error condition? Or does that indicate a configuration error?

from racecar.

Unfortunately it can be both: https://github.com/edenhill/librdkafka/wiki/FAQ#why-am-i-seeing-receive-failed-disconnected (via confluentinc/librdkafka#1664). However, the client misconfiguration is only happening when librdkafka and broker version are an incompatible match, which seems much less likely than random TCP hiccup.

So, something like this around polling?

begin
  # …
rescue => e
  retries = (retries || 0) + 1
  raise if e.error_code != 123456 || retries > MAX_RETRIES
  sleep 2 ** retries
  retry
end

Please note that we were generally letting Kubernetes handle these restarts for us, so we might have missed some error cases if they didn't occur regularly enough.

from racecar.

Racecar's contract with the user is that it doesn't crash on ephemeral problems – so I think we should do some kind of retries with exponential backoff.

from racecar.