Comments (18)
We already have so many, adding just for the sake of having all might in turn cause the configuration to be over complicated.
from azure-functions-kafka-extension.
I like your proposal. For Idempotence we need to check how we handle errors in our producer
from azure-functions-kafka-extension.
My suggestions would be for you to take a look at the librdkafka and Confluent.Kafka configuration docs and propose what settings should be exposed in the function.
I would start with settings that affect performance and throughput. Settings about max message size could be important too, in case we need to support topics with large messages.
At the end we should add the newly exposed options in our documentation section.
Does it help @brandonh-msft ?
from azure-functions-kafka-extension.
Are those consumer or producer related settings? It is unclear to me.
Regarding commit configuration: we are doing manual commits, therefore enable.auto.commit and auto.commit.interval.ms will not be used/exposed.
from azure-functions-kafka-extension.
Can someone please update the docs and send a PR for bullet point 2 then. sounds like it's done.
from azure-functions-kafka-extension.
To help in somebody picking this up, can we define "important" in this issue?
from azure-functions-kafka-extension.
perfect, thanks
from azure-functions-kafka-extension.
@brandonh-msft is this* something you're going to tackle?
- take a look at the librdkafka and Confluent.Kafka configuration docs and propose what settings should be exposed in the function.
At the end we should add the newly exposed options in our documentation section.
from azure-functions-kafka-extension.
you can put this on my backlog but i'm tackling #44 first
from azure-functions-kafka-extension.
Ok, assigned to you
from azure-functions-kafka-extension.
K so right now, looking at Global configuration properties in librdkafka docs I'm thinking:
property | proposed name | description |
---|---|---|
message.max.bytes | MaxMessageBytes | Maximum transmit message size. |
enable.auto.commit | AutoCommit | Automatically and periodically commit offsets in the background. |
auto.commit.interval.ms | AutoCommitIntervalMs | The frequency in milliseconds that the consumer offsets are commited (written) to offset storage. (0 = disable) |
batch.num.messages | BatchSize | Maximum number of messages batched in one MessageSet. |
It's not clear to me if some of those are implementations we'd have to code up (autocommit, for instance) or if it's something we specify on the msg that's sent and the protocol/receiver takes care of it. Would be good to get some clarity there if you guys know off the top of your head.
Nothing else is standing out as either applicable to sending messages, or particularly useful.
from azure-functions-kafka-extension.
yeah TBH i'm not sure either; i agree it's not super clear in the docs whether it's producer related or not. I just kinda guessed.
So now considering just:
property | proposed name | description |
---|---|---|
message.max.bytes | MaxMessageBytes | Maximum transmit message size. |
batch.num.messages | BatchSize | Maximum number of messages batched in one MessageSet. |
from azure-functions-kafka-extension.
Ok. let's start with these. and when people request others or want extra features that are unblocked by exposing others, then we can go add them.
from azure-functions-kafka-extension.
Not sure if we should expose:
enable.idempotence
source: https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md
from azure-functions-kafka-extension.
After some more digging found these are all the knobs even possible to set w/ the Kafka .Net SDK - do we just expose them all?
from azure-functions-kafka-extension.
fair enough - it doesn't look like, though, i can tweak the pieces I called out earlier after looking at what the .Net SDK exposes in ProducerConfig
- which of the above do you think we'd want to expose, then? Perhaps just the EnableIdempotence
? I can see value in the Timeouts, BatchNumMessages
, and MessageSendMaxRetries
, though.
from azure-functions-kafka-extension.
Yeah, go for the simplest config that enables the majority of scenarios. We can start exposing more config options as developers using this start requesting additional features. Too many knobs and dials get confusing
from azure-functions-kafka-extension.
@brandonh-msft What's the status of this? In progress? Or complete?
from azure-functions-kafka-extension.
Related Issues (20)
- Target scaler is not working (.NET) HOT 1
- Inconsistency between consumer and producer config HOT 1
- How to do exception handling in kafka output trigger if trigger fails to write to kafka HOT 2
- Distributed Tracing and kafka trigger
- Kafka trigger parameterization issues HOT 2
- Confluent Packages out of date causing runtime error
- Support for Bring your own certificates (.pfx)
- SchemaRegistryUrl attribute does not work with Java Azure Functions
- Enable support for rich datatypes in dotnet - isolated based apps using Kafka extension
- Kafka Trigger, Exception in Kafka subscriber, System.ObjectDisposedException at Confluent.Kafka.Impl.SafeKafkaHandle.ThrowIfHandleClosed
- Combine Key, Value, Header, TimeStamp in the same .Net object
- Make Kafka Output a first class typed citizen
- Support for librdkafka compression.codec and compression.type
- Kafka extension V3.9.0 issue HOT 1
- A lot of redundant logs produced HOT 1
- Azure Function (Java ) to Confluent Kafka output binding error
- Typescript Support for Functions v4?
- Upgrading .NET version for Extension Projects
- Confluent SDK upgrade
- Scale to Zero Support for Kafka Extension
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from azure-functions-kafka-extension.