System.Net.Sockets.SocketException - Dropping Messages? about azure-functions-eventhubs-extension HOT 9 CLOSED

The error comes from:
https://github.com/Azure/azure-functions-eventhubs-extension/blob/dev/src/Microsoft.Azure.WebJobs.Extensions.EventHubs/Listeners/EventHubListener.cs#L160

The method called when the underlying client experiences an error while receiving. EventProcessorHost will take care of recovering from the error and continuing to pump messages.
https://docs.microsoft.com/en-us/dotnet/api/microsoft.azure.eventhubs.processor.ieventprocessor.processerrorasync?view=azure-dotnet

So nothing to worry about, your messages did not drop as there wasn't checkpoint operation.

from azure-functions-eventhubs-extension.

Thanks @alrod

So if I understand correctly this is a bug. Because an expected exception (System.Net.Sockets.SocketException) is being logged with a high severity?

What's the best place to report/fix this?

from azure-functions-eventhubs-extension.

I am not sure if this a valid bug:
The method called when the underlying client experiences an error while receiving. EventProcessorHost will take care of recovering from the error and continuing to pump messages.
https://docs.microsoft.com/en-us/dotnet/api/microsoft.azure.eventhubs.processor.ieventprocessor.processerrorasync?view=azure-dotnet

Errors during pumping new messages are expected and EventHub guranties a message a will be delivered at least once.

How often do you get SocketExceptions ? only during a night?

from azure-functions-eventhubs-extension.

I guess the bug for me is the serverity (just like in #5).

Consider:

• It's expected. So it's not exceptional.
• If theres no action I can take (the event processor host recovers itself, and I don't think I even have access to it?) - then why do I need to know about it?

I am trying to monitor the event hub -> function app connection so that I can quickly react to any issues. For example, just today a company I'm working with revoked the shared access key, and but watching for errors I could quickly let them know and restore the string.

Having a couple of message with a high serveriy level where the only action we can take is to ignore them does not scalable. Especially when they happen at two in the morning!

from azure-functions-eventhubs-extension.

(To answer your question - we've had 16 over the last week)

from azure-functions-eventhubs-extension.

we can't change severity level for specific exception types. we consider ReceiverDisconnectedException and LeaseLostException as information because these exceptions can happen as part of normal partition balancing across instances.

16 per week is fairly low amount, I only can recommend adjust your alerting logic:

1. Filter by specific exception types.
2. Send an alert if you have more then x erros during y period of time.

from azure-functions-eventhubs-extension.

Ok, I guess I'm just missing context here

The issue with only sending an alert if there are more than x a minute is that we risk missing actual exceptions. (For example, we get the odd storage exception that happens due to an error we need to resolve).

And if I filter by the socket exception, then it feels like we're all acknowledging that all socket errors expected in normal operation; and that there's no risk to ignoring them?

I guess I'm just missing the context as to why we can't alter the severity of the exception that we know is not exceptional?

Maybe there is an opportunity to document a list of exceptions, like these, which are thrown as part of the internals of the function app when discussing monitoring event hub integrations?

(As an aside, 16 might not sound like much, but there's only 10 devices pushing data; which is why it doesn't feel very scalable)

from azure-functions-eventhubs-extension.

Or to put it another way. It feels like you're saying it's reasonable to expect anyone who wants to monitor an event hubs integration to build the following query:

exceptions
| where severityLevel > 1 // only alert on errors and warnings
| where innermostMessage != "The lease ID specified did not match the lease ID for the blob." // known bug: https://github.com/Azure/azure-functions-eventhubs-extension/issues/42
| where problemId != "System.Net.Sockets.SocketException at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw" // unexceptional: https://github.com/Azure/azure-functions-eventhubs-extension/issues/43
| where not (problemId startswith "System.ArgumentOutOfRangeException at Microsoft.Azure.EventHubs.Processor.PartitionContext") // unexceptional: https://stackoverflow.com/questions/51602965/azure-eventhub-function-leaselostexception

from azure-functions-eventhubs-extension.

Closing due to inactivity.

from azure-functions-eventhubs-extension.