Excessive overprovisioning with Event Hub Trigger about azure-functions-eventhubs-extension HOT 22 CLOSED

Hi @UnderShasha, I set the properties.functionAppScaleLimit to 2 and checked over the last week. And I indeed never saw more than two instances at a time. I will keep an eye on it, but this really seems to solve the issue.

from azure-functions-eventhubs-extension.

I got a hint from the support team, that I could try WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT = 2.

That seems to work and will limit the number of server instances.

That seems to reduce the number of servers somewhat (currently I see 7 instances).

from azure-functions-eventhubs-extension.

+@cgillum who is more of an expert on this than I am. Chris, this does seem like a large overprovisioning (up to 30 instances for 4 partitions). Is this a known issue?

It's odd that the error you get from EventHubs is Microsoft.WindowsAzure.Storage.StorageException: Operation could not be completed within the specified time. That seems like a timeout listing the blobs, rather than a Conflict that I would expect from a locked blob lease.

from azure-functions-eventhubs-extension.

@brettsam The EventProcessorHost used by Event Hubs trigger will periodically do a list blobs operation to determine whether the leases are evenly distributed.

But yes, this does sound like major over-provisioning. Are there any other functions besides this one in the app? Also, was VNET or runtime-driven scaling configured for this app?

from azure-functions-eventhubs-extension.

Also, @Peter-B- do you have a reference to the Microsoft support case that I can use to get more information about your app? I'd be interested to take a closer look at your case.

from azure-functions-eventhubs-extension.

Hi @cgillum, Thanks for your support.

I guess the reference number would be 120090824005307.

There is one other function in this app, which is deactivated using the [Disable()]-attribute. The event hub triggered function is the only one running.

There is no VNet configured.

I am not sure about the runtime-driven scaling. Where would I configure this?

from azure-functions-eventhubs-extension.

Hi @Peter-B-, thanks for bringing this to our attention. I've identified some faulty partitioning logic that resulted in egregious over-scaling for your app. I'm working on a fix for this in our private repository now. In the meantime, I've mitigated the over-scaling by setting the WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT app setting to the partition count (your app was showing a setting of "__ WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT __", which I assume was a mistake).

from azure-functions-eventhubs-extension.

Hi @glennamanns, thanks for addressing this issue.

The __ was intentional. As I wrote before, I set WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT = 2, but that didn't fix the problem. However, I got the feeling that it improved the situation, i.e. I found only 6 to 8 instances instead of 15 to 25.

I "removed" the setting by applying __ in an attempt to verify that finding before posting it here. But I didn't find time yesterday to check it.

I'm glad to hear that you found the cause of the problem and I'm looking forward to hearing from you.

from azure-functions-eventhubs-extension.

Hi,

I have similar observations with our deployment, where we witness 20-22 server instances for an EH with 5 partitions. Our support ticket with MSFT helped get clarification and I am referencing the ticket here as reference. Case Number 120091023001959
I will try the WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT setting

from azure-functions-eventhubs-extension.

Hi,
I am not sure if WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT setting works as expected. I have this set to 5 and I can see 12 servers online and the count keeps going up

from azure-functions-eventhubs-extension.

Hi,
Manual Stop and Start seems to have helped contain the scaling

from azure-functions-eventhubs-extension.

All of a sudden, server instances increased to 15. Manual start and stop doesn't help either
@glennamanns , any comments on this?

from azure-functions-eventhubs-extension.

Hi, are there any news on this topic?

from azure-functions-eventhubs-extension.

@Peter-B- , do you exp. similar issues with overprovisioning even with that recommended APP setting?

from azure-functions-eventhubs-extension.

Yes. This settings only mitigates the problem somewhat. It does not solve it.

Support told me that the AF team is working on a solution, but I didn't get a feedback here.

from azure-functions-eventhubs-extension.

@Peter-B- and @alkreddy, the fix is scheduled to be rolled out in the upcoming platform release.

@glennamanns - Is there any mitigation to avoid the over scaling of the instance?

However, it seems we have option to limit the scaling. Could you guys try to implement as suggested in below article?
https://docs.microsoft.com/en-us/azure/azure-functions/functions-scale#limit-scale-out

from azure-functions-eventhubs-extension.

@UnderShasha Is 'limit-scale-out' different than manually setting the App Configuration WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT?

from azure-functions-eventhubs-extension.

@alkreddy - Functionality is same but the settings are different. Could you try that out and see if you can limit the scaling.

from azure-functions-eventhubs-extension.

@UnderShasha , Same here. with properties.functionAppScaleLimit, the server scaling is limited to the property value

from azure-functions-eventhubs-extension.

@Peter-B- and @alkreddy - Good news. Thank you for the update.

from azure-functions-eventhubs-extension.

@UnderShasha hello, from what I understand you're from Microsoft. Can you please let us know when this is going to be fixed on the infrastructure or library level? The properties.functionAppScaleLimit, does reduce the amount of list calls drastically, but at the same it kills the very idea of automatic scalability which is important , in my case, for the scenarios with the unknown amount of data streamed, because the limit caps that scalability. This is a very essential part of functionality, and as above mentioned in the current state and without the cap the massive costs for inadequate overprovisioning (list and other operations) can be incurred. Is there a timeline on a sort of permanent solution of the problem that doesn't kill the flexibility of auto-scale?

from azure-functions-eventhubs-extension.

@pragnagopa, is this can be on scale controller side?

from azure-functions-eventhubs-extension.