Azure Function does not process events even when there is more than 1 partition about azure-functions-kafka-extension HOT 5 OPEN

@Vijay-Nirmal Can you please provide information on the number of instances running for the function app when you observed this behavior?
As the function app scales - each instance should be able to process the assigned partitions. This should lead to parallel processing of the partitions. //@krishna-kariya

from azure-functions-kafka-extension.

@jainharsh98 FUNCTIONS_WORKER_PROCESS_COUNT was already set as 10. Even with 10 it had the same behavior.

Worker count does not matter because on host you can have a situation when a buffer of messages is fulfiled by messages only from one partition, and only these messages will be procces and send to workers.

Check this KIP
https://cwiki.apache.org/confluence/display/KAFKA/KIP-41%3A+KafkaConsumer+Max+Records#KIP41:KafkaConsumerMaxRecords-EnsuringFairConsumption

Unfortunately, it is not the case that each worker is a separate consumer in Kafka.

from azure-functions-kafka-extension.

@jainharsh98 Only one instance (reason is below), but we need to process multiple events in parallel we don't care about order. Is it not possible with one instance?

The reason we have only one instance (for now) is because we are using Dedicated Plan (App Service Plan), in that there is no proper metric to perform auto-scaling.

from azure-functions-kafka-extension.

@Vijay-Nirmal If you are working with a single instance - I would suggest you try increasing the Worker Process Count as described here depending on the number of cores provided by your plan and the load being processed.
For dedicated plan you can have both autoscaling and set the number of instances you want manually - In case you want to use scaling for parallel processing of partitions on separate instances.

from azure-functions-kafka-extension.

@jainharsh98 FUNCTIONS_WORKER_PROCESS_COUNT was already set as 10. Even with 10 it had the same behavior.

from azure-functions-kafka-extension.