Must have all configuration and defaults conforming to Functions conventions about azure-functions-kafka-extension HOT 11 CLOSED

@jeffhollan can you give a quick review of the config exposed, as we're not exposing ALL kafka config and also the defaults we've chosen.

from azure-functions-kafka-extension.

Do you have an example of some of the KafkaOptions or similar? In general we try to model the host.json options off of the options exposed, and maybe only allow sub-set to be settable. Here's the event hubs options as a starter:

https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-event-hubs#host-json

from azure-functions-kafka-extension.

Here's the KafkaOptions are they are today - https://github.com/Microsoft/azure-functions-kafka-extension/blob/master/src/Microsoft.Azure.WebJobs.Extensions.Kafka/KafkaOptions.cs

from azure-functions-kafka-extension.

ReconnectBackoffMs
ReconnectBackoffMaxMs
StatisticsIntervalMs
SessionTimeoutMs
MaxPollIntervalMs
QueuedMinMessages
QueuedMaxMessagesKbytes
MaxPartitionFetchBytes
FetchMaxBytes
MaxBatchSize
SubscriberIntervalInSeconds
ExecutorChannelCapacity
ChannelFullRetryIntervalInMs

from azure-functions-kafka-extension.

@fbeltrao are all these used by the code? also, I see from the comments that these each have defaults, are these defaults coming from the confluent driver, or have we set them ourselves somewhere?

from azure-functions-kafka-extension.

Would be interested to know which of these users may want control on but my initial thinking was all the Ms or Seconds options we just manage and expose all the bytes and size options. But not against just surfacing all of these as overrides in host.json with intelligent defaults

from azure-functions-kafka-extension.

The following are exposed by Confluent:

ReconnectBackoffMs
ReconnectBackoffMaxMs
StatisticsIntervalMs
SessionTimeoutMs
MaxPollIntervalMs
QueuedMinMessages
QueuedMaxMessagesKbytes
MaxPartitionFetchBytes
FetchMaxBytes

I use their defaults, which you can see here (https://github.com/confluentinc/confluent-kafka-dotnet/blob/v1.0-beta3/src/Confluent.Kafka/Config_gen.cs)

Those are ours:
MaxBatchSize: largest KafkaEventData[] we will collect before executing function
SubscriberIntervalInSeconds: interval in which data will be send to function, only if you don't reach MaxBatchSize
ExecutorChannelCapacity: amount of batches that can be pending before the Kafka Subscriber pauses.
ChannelFullRetryIntervalInMs: interval in which we verify that the function is catching up with the demand in order to resume the subscriber

from azure-functions-kafka-extension.

So should we "hide" the Confluent ones and only expose ours?
Or just expose (and document) them all?

from azure-functions-kafka-extension.

I'd like to give freedom to those that know what they are doing. On the other hand, a miss configuration can make the service unusable, and the responsibility lies on us when hosting in Azure. An option would be to give more freedom when hosting the function outside Azure or in Service Plan, while remaining restrictive in a consumption plan (once/if we get there).

For instance:
FetchMaxBytes, MaxPartitionFetchBytes and QueuedMaxMessagesKbytes might be important on a scenario where the Kafka message is large, but it might require more memory from the host.

from azure-functions-kafka-extension.

I'm fine to expose all for now and just make sure we document behavior and defaults. We may regret it and change that in the future, but I'm ok providing the flexibility to those who may be able to take advantage of it for now

from azure-functions-kafka-extension.

Ok. Will close this issue as complete then. Let's make sure we document the defaults and behaviour of each config setting.

from azure-functions-kafka-extension.