Support sync interface about librdkafka HOT 6 CLOSED

Can you tell me about your use case for this?
Do you want to send one or multiple messages in each blocking synchronic produce_sync() call?

from librdkafka.

Hi,
we want to do the fail-tolerant operation, when the kafka cluster is not available. Currently there is message deliver callback. but some case the fail-tolerant will process twice. so we ask for sync interface.

I think you can give choice for log count sending per call.

Thanks~!

from librdkafka.

What do you mean by "fail-toerant will process twice"?
Do you mean the message send retry that librdkafka performs if the broker connection goes down?
The retry count is configurable through message.send.max.retries which defaults to 2.

from librdkafka.

fail-tolerant is implement in our program. if send the log to kafka failed, we should process it in our environment. and when the kafka is recover, we will send failed log to kafka again.
if we use async interface it is not good for this. we must wait the deliver callback method is callback, and do such fail-tolerant things. yesterday , I already find a way to avoid duplicated process failed logs message. use the last parameter of rd_kafka_produce().

but I think sync interface is easy for user to deal with this situation.

Thanks~!

from librdkafka.

I understand.
The drawbacks of a sync interface is that only a small number of messages may be produced at a given time and the application must queue messages for producing. The throughput is also minimal, at best, and is adversely affected by the application <-> broker latency.
Having said that I will look in to making a sync producer interface.

The current async produce interface provides full message tracking and reliability.
For each message produced the application may provide a pointer (that typically points to some application per-message object) called msg_opaque.
This msg_opaque value is provided to the application in the delivery report (dr) callback, along with an error code (to indicate success or specific failures), thus allowing the application to take actions if delivery of a message fails.

Read this for more information:
https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#producer-api

The async produce interface also offers high performance and is not particularily affected by a high latency between application and broker.

from librdkafka.

See this wiki page for how to implement a sync producer in your application:
https://github.com/edenhill/librdkafka/wiki/Sync-producer

from librdkafka.