Should have native .NET output binding support about azure-functions-kafka-extension HOT 18 CLOSED

We already have so many, adding just for the sake of having all might in turn cause the configuration to be over complicated.

from azure-functions-kafka-extension.

I like your proposal. For Idempotence we need to check how we handle errors in our producer

from azure-functions-kafka-extension.

My suggestions would be for you to take a look at the librdkafka and Confluent.Kafka configuration docs and propose what settings should be exposed in the function.

I would start with settings that affect performance and throughput. Settings about max message size could be important too, in case we need to support topics with large messages.

At the end we should add the newly exposed options in our documentation section.

Does it help @brandonh-msft ?

from azure-functions-kafka-extension.

Are those consumer or producer related settings? It is unclear to me.
Regarding commit configuration: we are doing manual commits, therefore enable.auto.commit and auto.commit.interval.ms will not be used/exposed.

from azure-functions-kafka-extension.

Can someone please update the docs and send a PR for bullet point 2 then. sounds like it's done.

from azure-functions-kafka-extension.

To help in somebody picking this up, can we define "important" in this issue?

from azure-functions-kafka-extension.

perfect, thanks

from azure-functions-kafka-extension.

@brandonh-msft is this* something you're going to tackle?

• take a look at the librdkafka and Confluent.Kafka configuration docs and propose what settings should be exposed in the function.
  At the end we should add the newly exposed options in our documentation section.

from azure-functions-kafka-extension.

you can put this on my backlog but i'm tackling #44 first

from azure-functions-kafka-extension.

Ok, assigned to you

from azure-functions-kafka-extension.

K so right now, looking at Global configuration properties in librdkafka docs I'm thinking:

It's not clear to me if some of those are implementations we'd have to code up (autocommit, for instance) or if it's something we specify on the msg that's sent and the protocol/receiver takes care of it. Would be good to get some clarity there if you guys know off the top of your head.

Nothing else is standing out as either applicable to sending messages, or particularly useful.

from azure-functions-kafka-extension.

yeah TBH i'm not sure either; i agree it's not super clear in the docs whether it's producer related or not. I just kinda guessed.

So now considering just:

from azure-functions-kafka-extension.

Ok. let's start with these. and when people request others or want extra features that are unblocked by exposing others, then we can go add them.

from azure-functions-kafka-extension.

Not sure if we should expose:
enable.idempotence

source: https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md

from azure-functions-kafka-extension.

After some more digging found these are all the knobs even possible to set w/ the Kafka .Net SDK - do we just expose them all?

from azure-functions-kafka-extension.

fair enough - it doesn't look like, though, i can tweak the pieces I called out earlier after looking at what the .Net SDK exposes in ProducerConfig - which of the above do you think we'd want to expose, then? Perhaps just the EnableIdempotence? I can see value in the Timeouts, BatchNumMessages, and MessageSendMaxRetries, though.

from azure-functions-kafka-extension.

Yeah, go for the simplest config that enables the majority of scenarios. We can start exposing more config options as developers using this start requesting additional features. Too many knobs and dials get confusing

from azure-functions-kafka-extension.

@brandonh-msft What's the status of this? In progress? Or complete?

from azure-functions-kafka-extension.