Event Hubs extension for Azure Functions
License: MIT License
This extension provides functionality for receiving Event Hubs messges in Azure Functions, allowing you to easily write functions that respond to any message published to Event Hubs.
| Branch | Status |
|---|---|
| dev |
This project is under the benevolent umbrella of the .NET Foundation and is licensed under the MIT License
This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.
![microsoft-github-policy-service[bot] avatar](https://avatars.githubusercontent.com/in/95686?v=4 "microsoft-github-policy-service[bot]")
I found a typo in the EventHubTriggerInput.cs file. It should be "Partition" instead of "Partion".
This leads to a typo when it gets logged to App Insights as well:
path: src/Microsoft.Azure.WebJobs.Extensions.EventHubs/Triggers/EventHubTriggerInput.cs
When using the EventHubTrigger with batched events, we are observing that the position on the stream is advanced when the Function times out. This is in the consumption plan.
Per the doc here: https://docs.microsoft.com/en-us/azure/azure-functions/functions-reliable-event-processing
If conditions prevent the function execution from completing, the host fails to progress the pointer. If the pointer isn't advanced, then later checks end up processing the same messages.
I would expect that the function timing out is a condition that prevented function execution from completing.
Provide the steps required to reproduce the problem
Create a Function with an EventHubTrigger, and a code loop that will cause a timeout.
Add some items to the EventHub for the function to process.
Upon timeout, the next function invocation should pull from the same position as the previous invocation.
The position in the stream has been advanced, and only new events will be processed.
As per the documentation on storage partitioning, The partition server where a blob would be processed is determined by the partition key generated for the blob.
Each blob has a partition key comprised of the full blob name (account+container+blob). The partition key is used to partition blob data into ranges. The ranges are then load-balanced across Blob storage. Range-based partitioning means that naming conventions that use lexical ordering (for example, mypayroll, myperformance, myemployees, etc.) or timestamps (log20160101, log20160102, log20160102, etc.) are more likely to result in the partitions being co-located on the same partition server.โ
The container name for the event hub and function host start with the same prefix. So it is possible that all the blob operations are happening on the same partition server, increasing the chance of partition rebalancing operation.
Repro steps
Customers use the Event Hub Extensions bindings very frequently.
If these don't work then we need to ask the customer for the details of those Event Hubs and the related Storage account to see if there is an issue on that side.
Adding this information when the Event Hub extension starts to listen on the Event Hub extensions would allow us to be able to
a) Build Detectors that access this information if the customer does raise a support request.
b) Look at automated solutions that will check connectivity to the EventHub namespace and allow the customer to resolve transient issues on their own.
Expected behavior
A log entry is written when we attach to the binding.
Actual behavior
We log information about the Event Hub and Partition Count but not the FQDN of the Event Hub Namespace.
Known workarounds
No workarounds exist.
My project using the EventHub Trigger Function. The WebJobs.Extensions.EventHubs SDK version is 3.0.6
Now I'm experiencing the LeaseLostException in App Insight, not frequent.
Call Stack
Microsoft.Azure.EventHubs.Processor.LeaseLostException:
at Microsoft.Azure.EventHubs.Processor.AzureStorageCheckpointLeaseManager+<AcquireLeaseCoreAsync>d__32.MoveNext (Microsoft.Azure.EventHubs.Processor, Version=2.1.0.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.Azure.EventHubs.Processor.PartitionManager+<RunLoopAsync>d__12.MoveNext (Microsoft.Azure.EventHubs.Processor, Version=2.1.0.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)
Inner exception Microsoft.WindowsAzure.Storage.StorageException handled at Microsoft.Azure.EventHubs.Processor.AzureStorageCheckpointLeaseManager+<AcquireLeaseCoreAsync>d__32.MoveNext:
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor+<ExecuteAsyncInternal>d__41.MoveNext (Microsoft.WindowsAzure.Storage, Version=9.3.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.Azure.EventHubs.Processor.AzureStorageCheckpointLeaseManager+<AcquireLeaseCoreAsync>d__32.MoveNext (Microsoft.Azure.EventHubs.Processor, Version=2.1.0.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)
I understood this is not a real error after reading a lot of doc. I'm very glad to see you guys want to log it as Information in #5 and #42 .
Could you guys please also update the LogExceptionReceivedEvent function in EventHubExtensionConfigProvider, your code logger?.Log(logLevel, 0, message, e.Exception, (s, ex) => message); still logging the LeaseLostException as Exception - https://github.com/Azure/azure-functions-eventhubs-extension/blob/dev/src/Microsoft.Azure.WebJobs.Extensions.EventHubs/Config/EventHubExtensionConfigProvider.cs#L87
Issue Summary
Customers using the Azure Functions Event Hub Trigger may see that the Event Hub fails to trigger.
This happens when the Function App has AzureWebJobsStorage set to use a Storage v2 Account and Data Lake Storage Gen2 Hierarchical Namespace Enabled.
Issue Details
In addition if you check the Storage account you may notice that the leases have expired on the containers for the Consumer Group and Partition Id.
Or in Storage Explorer
The Azure Functions Event Hub Trigger uses the Event Processor Host library version 3.0.0. which uses v9 of the Storage SDK.
I think it might be related to the hierarchical namespace support that gets added from ADLS Gen 2. The leases are taken out on {EventHubNamespace}/{EventHubName}/{ConsumerGroupName}/{PartitionNumber} .
Issue Mitigation
There are currently two mitigations that will work longer term.
Sample Code
This code may need modifications as this is dependent on the number of partitions that the Event Hub supports.
`
using System;
using System.IO;
using System.Text;
using System.Threading.Tasks;
using Azure.Storage.Blobs;
using Microsoft.Azure.EventHubs;
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Logging;
using Azure.Storage.Blobs.Specialized;
using Azure.Storage.Blobs.Models;
using Azure;
namespace TestLeaseRenewal
{
public static class ExpiredLeaseFixer
{
[FunctionName("ExpiredLeaseFixer")]
public static async Task Run(
[TimerTrigger("0 */5 * * * * ")] TimerInfo myTimer,
ILogger log)
{
string connectionString = "{useStorageAccountConnStrHere";
string containerName = "azure-webjobs-eventhub";
string[] blobNames = {
"/{eventHubNameSpace}.servicebus.windows.net/{EventHubName}/{ConsumerGroupName}/0",
"/{eventHubNameSpace}.servicebus.windows.net/{EventHubName}/{ConsumerGroupName}/1",
"/{eventHubNameSpace}.servicebus.windows.net/{EventHubName}/{ConsumerGroupName}/2",
"/{eventHubNameSpace}.servicebus.windows.net/{EventHubName}/{ConsumerGroupName}/3",
"/{eventHubNameSpace}.servicebus.windows.net/{EventHubName}/{ConsumerGroupName}/4",
};
BlobServiceClient blobServiceClient = new BlobServiceClient(connectionString);
BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient(containerName);
foreach (string blobName in blobNames)
{
BlobClient blobClient = containerClient.GetBlobClient(blobName);
BlobLeaseClient blobLeaseClient = new BlobLeaseClient(blobClient);
try
{
BlobLease blobLease = await blobLeaseClient.AcquireAsync(TimeSpan.FromSeconds(15));
blobLease = blobLeaseClient.Break();
log.LogInformation("Lease for blob broken:" + blobName);
}
catch (Exception e)
{
log.LogInformation("Failed to break blob lease for:" + blobName);
}
}
}
}
}
`
Hi,
The Microsoft.Azure.WebJobs.Extensions.Storage package just got a major update to 4.0.0, which moves away from the old Microsoft.WindowsAzure.Storage APIs and now uses the latest Microsoft.Azure.Storage libraries.
Is there a similar update coming for Microsoft.Azure.WebJobs.Extensions.EventHubs?
I use both the extensions in the same project, and they are taking different downstream dependencies for the storage SDK. It would be good to have them aligned.
Also, if I try to update the Storage extension anyway, I get an error at runtime:
[5/28/2020 11:25:15 PM] EventProcessorHost error (Action=Checking Leases, HostName=32b60314-146f-4ebe-be2d-113f1a7c6b05, PartitionId=1)
[5/28/2020 11:25:15 PM] Microsoft.WindowsAzure.Storage: Server encountered an internal error. Please try again after some time.
By default max message is 256ะB what corresponds to basic plan. We need to add ability to overwrite default value in host.json settings as standard plan has 1MB max message size.
Related: #8
A recent Azure update (around 6/9) causes the function to stop executing on new event hub messages. We're currently on version 4.1.1, and we've found that deleting either the consumer group or the checkpoint files in Blob Storage cause the function to start processing everything stored in the event hub (as one might expect).
We've been using v4.1.1 for months before this happened, so our assumption is that some update to the Azure Functions Host or Azure Storage Accounts is causing this.
We found that downgrading from 4.1.1 to 3.0.6 fixes the issue.
Below is our project file with packages:
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>netcoreapp3.1</TargetFramework>
<AzureFunctionsVersion>v3</AzureFunctionsVersion>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Azure.Identity" Version="1.1.1" />
<PackageReference Include="Microsoft.Azure.Functions.Extensions" Version="1.0.0" />
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.EventHubs" Version="4.1.1" />
<PackageReference Include="Microsoft.NET.Sdk.Functions" Version="3.0.7" />To fix this we simply downgraded the project to version 3.0.6:
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>netcoreapp3.1</TargetFramework>
<AzureFunctionsVersion>v3</AzureFunctionsVersion>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Azure.Identity" Version="1.1.1" />
<PackageReference Include="Microsoft.Azure.Functions.Extensions" Version="1.0.0" />
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.EventHubs" Version="3.0.6" />
<PackageReference Include="Microsoft.NET.Sdk.Functions" Version="3.0.7" />I tried IoT Hub output binding from Node.js function as following documentation.
https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-event-iot-output?tabs=javascript
But following error shows and the function is failed.
Unauthorized access. 'Send' claim(s) are required to perform this operation
It seems that a connection string, which starts with Endpoint=sb:// ... and was set when adding the binding from portal, doesn't have enough permission.
Is this binding supported currently? # documentation says this is supported.
If this is supported, could you please let me know how to get appropriate connection string?
# As long as I know, EventHubs extention is used for IoT Hub binding, so I filed to this repo. Please let me know if this is wrong.
ServiceBus trigger adds logs for trigger execution which are useful for CRIs investigations:
https://github.com/Azure/azure-functions-servicebus-extension/blob/4c4ef3252112bc822a5c80a26824b12146330650/src/Microsoft.Azure.WebJobs.Extensions.ServiceBus/Triggers/ServiceBusTriggerInput.cs#L67
We need something similar for EventHub trigger.
A customer reported some issues when writing to Event Hub for their function app. The exception points to SendBatchAsync.
After reaching out to the Event Hub team, they mentioned that this could be caused by a networking issue, so adding retry logic will help mitigate this. The fix has to go in partitioned-output branch.
Here is the full stack trace:
Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: <FunctionName> ---> System.InvalidOperationException : Error while handling parameter _binder after function returned: ---> System.TimeoutException : The operation did not complete within the allocated time 00:00:00 for object out-connection82.
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.Azure.Amqp.AsyncResult.End[TAsyncResult](IAsyncResult result)
at Microsoft.Azure.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result)
at Microsoft.Azure.Amqp.AmqpObject.EndOpen(IAsyncResult result)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar,Func`2 endFunction,Action`1 endAction,Task`1 promise,Boolean requiresSynchronization)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.EventHubs.Amqp.AmqpEventHubClient.CreateConnectionAsync(TimeSpan timeout)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.Amqp.FaultTolerantAmqpObject`1.OnCreateAsync[T](TimeSpan timeout)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.Amqp.Singleton`1.CreateValue[TValue](TaskCompletionSource`1 tcs,TimeSpan timeout)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.Amqp.Singleton`1.GetOrCreateAsync[TValue](TimeSpan timeout)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.EventHubs.Amqp.AmqpEventDataSender.CreateLinkAsync(TimeSpan timeout)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.Amqp.FaultTolerantAmqpObject`1.OnCreateAsync[T](TimeSpan timeout)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.Amqp.Singleton`1.CreateValue[TValue](TaskCompletionSource`1 tcs,TimeSpan timeout)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.Amqp.Singleton`1.GetOrCreateAsync[TValue](TimeSpan timeout)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.EventHubs.Amqp.AmqpEventDataSender.OnSendAsync(IEnumerable`1 eventDatas,String partitionKey)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.EventHubs.Amqp.AmqpEventDataSender.OnSendAsync(IEnumerable`1 eventDatas,String partitionKey)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.EventHubs.EventHubClient.SendAsync(??)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.EventHubs.EventHubAsyncCollector.SendBatchAsync(String partitionKey,IEnumerable`1 batch)
Unlike Storage Queue trigger, EventHub trigger does not allow us to set maxPollingInterval. Is this intentional? If not, is this some thing that could be added? For e.g. it is configurable in logic apps with How often do you want to check for items field.
This is a continuation from SO thread.
Azure Functions on a consumption plan are automatically scaled. This can even lead to overprovisioning, as described in the docs. This means, that there are more Azure Function instances than Event Hub partitions. This is expected behavior.
However, I am experiencing a massive overprovisioning of up to 30 Azure Function Instances for 4 partitions. Those instances try to get a lease on a blob in the assigned storage account, which will fail for 26 of the 30 instances with a timeout.
These failed attempts to get a lease on the blobs causes significant costs for โlrs list and create container operationsโ, as well as for logged exceptions in Application Insights.
I had a discussion with Shashank from Microsoft Support and he reviewed some internal logs. There he found that
โข new instances are created, since messages queue up
โข instances are removed, since there are more instances than partitions
With this information, I believe that I encounter some issue with the scaling logic here:
Create an IoT Hub with 4 partitions and send messages to it from multiple devices
Create an Azure Function with an Event Hub Trigger
Consume messages in batches and add some Task.Delay(), so that batches have more than one message.
Let it run and check the number of โservers onlineโ in Application Insights.
Since the IoT Hub has 4 partitions, I would expect to see 1-4 Azure Function instances. I would even expect a mild overprovisioning of maybe 6 instances.
20 to 30 servers online and a lot of exceptions:
Microsoft.WindowsAzure.Storage.StorageException: Operation could not be completed within the specified time.
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)
at Microsoft.WindowsAzure.Storage.Blob.CloudBlobContainer.ListBlobsSegmentedAsync(String prefix, Boolean useFlatBlobListing, BlobListingDetails blobListingDetails, Nullable`1 maxResults, BlobContinuationToken currentToken, BlobRequestOptions options, OperationContext operationContext, CancellationToken cancellationToken)
at Microsoft.Azure.EventHubs.Processor.AzureStorageCheckpointLeaseManager.GetAllLeasesAsync()
at Microsoft.Azure.EventHubs.Processor.PartitionManager.RunLoopAsync(CancellationToken cancellationToken)
Stop the Azure Function, wait until all instances are killed and restart it. Then the problem is gone for some time (hours to days). I.E. less than 8 instances are created.
"Microsoft.Azure.WebJobs.Extensions.EventHubs" Version="4.1.1""Microsoft.NET.Sdk.Functions" Version="3.0.9"Although the trigger allows us to get the offset of the incoming message, we cannot set the offset of the desired message. This makes testing messages either from the IoTHub or Function side challenging, it also means that reprocessing messages because of a bad function or skipping messages because earlier messages do not need to be processed (or the data from IoTHub is bad) is not possible. Further, there may be a scenario when an IoT device comes online and I want to skip the first X number of messages because only the most recent one is the one I actually need to process. Adding this parameter helps control cost, compute and test scenarios.
Ex.
I create a new function and I have a full day of messages. I know that the first 1,000 messages will not work with this function so I want to skip over them without using compute/being charged for the fnc processing those 1,000. In fact, it may throw an error on the first 1,000.
Second, although a new consumer group may allow me to begin at the start the beginning, I should not need to always create new consumer groups (ex. I may want to always use $Default) for this scenario.
Create an Azure Function with the EventHub or IoTHub trigger and attempt to choose a specific EventHub offset.โ
Problem Statement: The partner knows that the first set of messages coming into a function from IoT Hub/ Event Hub should be ignored, so they want to be able to skip those messages. However, Azure Functions does not allow partners to skip messages in this way, and will charge the partner regardless.
Desired Experience: Partners should be able to set an offset for messages coming from Event Hub/IoT Hub.
#### Actual behavior
It is configured to receive and process messages detected from ServiceBus with Function in Azure environment.
When I checked Application Insights, I found a Microsoft.WindowsAzure.Storage.StorageException.
I want to know the cause of this phenomenon
A similar issue is registered, but it has not been resolved.
"The lease ID specified did not match the lease ID for the blob crashes worker."
Azure/azure-webjobs-sdk#2103
#### Repro steps
The reproduction procedure is not known because it has occurred only once.
Message:
The lease ID specified did not match the lease ID for the blob. The lease ID specified did not match the lease ID for the blob.
Exception type:
Microsoft.WindowsAzure.Storage.StorageException
Failed method:
Microsoft.WindowsAzure.Storage.Core.Executor.Executor+d__4`1.MoveNext
Call stack:
Microsoft.Azure.EventHubs.Processor.LeaseLostException:
at Microsoft.Azure.EventHubs.Processor.AzureStorageCheckpointLeaseManager+<RenewLeaseCoreAsync>d__34.MoveNext (Microsoft.Azure.EventHubs.Processor, Version=2.1.0.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.Azure.EventHubs.Processor.AzureStorageCheckpointLeaseManager+<UpdateLeaseCoreAsync>d__38.MoveNext (Microsoft.Azure.EventHubs.Processor, Version=2.1.0.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.Azure.EventHubs.Processor.AzureStorageCheckpointLeaseManager+<UpdateCheckpointAsync>d__18.MoveNext (Microsoft.Azure.EventHubs.Processor, Version=2.1.0.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.Azure.EventHubs.Processor.PartitionContext+<PersistCheckpointAsync>d__40.MoveNext (Microsoft.Azure.EventHubs.Processor, Version=2.1.0.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.Azure.WebJobs.EventHubs.EventHubListener+EventProcessor+<Microsoft-Azure-WebJobs-EventHubs-EventHubListener-ICheckpointer-CheckpointAsync>d__16.MoveNext (Microsoft.Azure.WebJobs.EventHubs, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.EventHubs, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Extensions.EventHubs\Listeners\EventHubListener.csMicrosoft.Azure.WebJobs.EventHubs, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 217)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.Azure.WebJobs.EventHubs.EventHubListener+EventProcessor+<CheckpointAsync>d__13.MoveNext (Microsoft.Azure.WebJobs.EventHubs, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.EventHubs, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Extensions.EventHubs\Listeners\EventHubListener.csMicrosoft.Azure.WebJobs.EventHubs, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 184)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.Azure.WebJobs.EventHubs.EventHubListener+EventProcessor+<ProcessEventsAsync>d__12.MoveNext (Microsoft.Azure.WebJobs.EventHubs, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.EventHubs, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Extensions.EventHubs\Listeners\EventHubListener.csMicrosoft.Azure.WebJobs.EventHubs, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 176)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.Azure.EventHubs.Processor.PartitionPump+<ProcessEventsAsync>d__30.MoveNext (Microsoft.Azure.EventHubs.Processor, Version=2.1.0.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)
Microsoft.Azure.EventHubs.Processor.AzureStorageCheckpointLeaseManager+<RenewLeaseCoreAsync>d__34.MoveNext ใงๅฆ็ใใใๅ
้จไพๅค Microsoft.WindowsAzure.Storage.StorageException:
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor+<ExecuteAsyncInternal>d__4`1.MoveNext (Microsoft.WindowsAzure.Storage, Version=9.3.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.Azure.EventHubs.Processor.AzureStorageCheckpointLeaseManager+<RenewLeaseCoreAsync>d__34.MoveNext (Microsoft.Azure.EventHubs.Processor, Version=2.1.0.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c)
#### Package version
Microsoft.Azure.WebJobs.ServiceBus 3.0.0-beta8
.NET Standard 2.0
Hello!
We are using an Azure Function to receive and distinct each message based on a Property on the message. The message should then be sent out to two different EventHubs (which is set as output binding attributes) based on the Property.
Steps to reproduce the issue:
Create an Azure Function that will trigger on EventHub message
Output the messages further to two different EventHubs based on a message property on the EventData input.
Expected behaviour is that these messages are being routed to each respective EventHub.
The EventHub output as configured "first" in the Run() parameter is receiving all events.
As a proof of this issue, I'm providing you with an example project, containing the core routing functionality we are trying to implement.
It contains of an Azure Function that triggers on an EventHub and then sends them further using the IAsyncCollector. The project also contains two additional Azure Functions which is triggered on the two output EventHubs. The result is that only the first Azure Function is being triggered (the one called EH1Receiver)
Link to example project: https://github.com/danielhertzman/eventhub-trigger-example
Hello Team,
We have enabled the EventHub with DR.
And in Azure function(v2 - core 2.1) we use Alias EventHub connection string to push the messages to the DR Enabled Eventhub.
What we noticed is that the response time for the FlushAsync() method is ~20ms higher than the EVenthub that don't have the DR Enabled.
So, just wondering if this is expected performance degradation given DR has to push the metadata to another region? or I'm missing some thing.
Any help on this one would be much appreciated. Please shout if you need any more info.
Thanks!
The dll released with Microsoft.Azure.WebJobs.Extensions.EventHubs 4.0.0 is not strongly named which is causing issues when we reference it. Could you please release a strongly named version of this file?
System.AggregateException
HResult=0x80131500
Message=One or more errors occurred.
Source=mscorlib
StackTrace:
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) in f:\dd\ndp\clr\src\BCL\system\threading\Tasks\Task.cs:line 2155
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification) in f:\dd\ndp\clr\src\BCL\system\threading\Tasks\Future.cs:line 567
at System.Threading.Tasks.Task`1.get_Result() in f:\dd\ndp\clr\src\BCL\system\threading\Tasks\Future.cs:line 532
at ...
Inner Exception 1:
FileLoadException: Could not load file or assembly 'Microsoft.Azure.WebJobs.EventHubs, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. Strong name validation failed. (Exception from HRESULT: 0x8013141A)
Inner Exception 2:
SecurityException: Strong name validation failed. (Exception from HRESULT: 0x8013141A)When looking at the dll directly with sn.exe I can see the following:
c:\>sn -vf Microsoft.Azure.WebJobs.EventHubs.dll
Microsoft (R) .NET Framework Strong Name Utility Version 4.0.30319.0
Copyright (c) Microsoft Corporation. All rights reserved.
Microsoft.Azure.WebJobs.EventHubs.dll is a delay-signed or test-signed assembly
It's the same issue for Microsoft.Azure.WebJobs.ServiceBus.dll released in Microsoft.Azure.WebJobs.Extensions.ServiceBus 3.1.0
Regards
Kristoffer
Repro steps:
[9/3/2019 8:03:15 PM] System.Private.CoreLib: Exception while executing function: Functions.EventHubTrigger. Microsoft.Azure.WebJobs.Host: Exception binding parameter 'eventHubMessages'. Microsoft.Azure.WebJobs.Host: Binding parameters to complex objects (such as 'Object
') uses Json.NET serialization.
1. Bind the parameter type as 'string' instead of 'Object' to get the raw values and avoid JSON deserialization, or
2. Change the queue payload to be valid json. The JSON parser failed: Unexpected character encountered while parsing value: a. Path '', line 0, position 0.
Issue does not happen if we write a JSON (e.g. { "b" : "a" } to the EventHub or specify dataType as string in the function.json.
Default behavior should be just to have the string without any deserialization?
We need to dispose EventHubProcessor on EventHubListener disposing. The current behavior can lead to processing messages after EventHubListener is disposed.
Hi,
I am facing the below error when trying to start my azure functions. The runtime for my azure functions is .netstandard 2.0.
My scenario is this: I have a common library(.netstandard) the functions has an EventHubTrigger and all was working well.
After I add "Microsoft.Azure.Eventhubs" to the common library consumed by my function the error starts occurring. The reason I need this library is because the common library posts to EH directly (not using triggers)
Downgrading to version Microsoft.Azure.EventHubs 3.0.0 is a workaround and resolves the issue.
The listener for function 'xxx' was unable to start.
[13-Aug-19 16:27:51] The listener for function 'xxx' was unable to start. Microsoft.Azure.EventHubs.Processor: Encountered error while fetching the list of EventHub PartitionIds. Microsoft.Azure.EventHubs.Processor: Method not found: 'Microsoft.Azure.EventHubs.EventHubClient Microsoft.Azure.EventHubs.EventHubClient.Create(System.Uri, System.String, Microsoft.Azure.EventHubs.ITokenProvider, System.Nullable`1<System.TimeSpan>, Microsoft.Azure.EventHubs.TransportType)'.
This is because Microsoft.Azure.WebJobs.Extensions.EventHubs depends on version 3.0.0 of Microsoft.Azure.EventHubs, but other projects are using version 4.1.0.
Causing the above conflict.
This is a request to update Microsoft.Azure.WebJobs.Extensions.EventHubs to use Version 4.1.0 of Microsoft.Azure.EventHubs
Thankyou
currently i have problem using IAsyncCollector in order to send a message to Event hub with a partition key. how is possible to choose a partition key?
Sample Code:
[EventHub("myevent", Connection = "connectionstring")] IAsyncCollector outboundMessage)
is there any option to choose the partition key by using the following method?
await outboundMessage.AddAsync(message);
Thank you.
Ehsan
Function version ~3
Using Microsoft.Azure.WebJobs.Extensions.EventHubs version 4.1.0
The nuget has a dependency on Microsoft.Azure.EventHubs.Processor 3.0.0 and by default restores and loads 3.0.0
Workaround:
Referencing Microsoft.Azure.EventHubs.Processor 4.1.0 solved the issue
The following two tests fail intermittently:
In EventHub_MultipleDispatch_BatchSend, the test fail when the OperationId is compared. This only happens a single message is dispatched from Event Hubs v/s a batch of messages. It's non deterministic if Event Hubs will dispatch a single message or more than one message at a time as it might depend on timing in the backend. This explains why the failures are intermittent.
In EventHub_MultipleDispatch_IndependentMessages the test fails when the parent operation Id is expected to be null but is not.. In this case, there is no logic checking if one ore more than one messages were dispatched to the function handler and I believe the test fails when there is just one message as the Parent
Some improvements and fixes with the latest version of the event processor host we should look at integrating into our trigger
https://github.com/Azure/azure-event-hubs-dotnet/releases/tag/3.0.0
We currently use 2.1.0
A few other fixes we'll want to group in this:
(I'm still finding my way around Azure, so apologise if this is not the approrpiate place to raise this issue).
We have a Node.js Function App configured with an Azure Event Hubs trigger.
Because this is a fairly critical piece of infrastructure to us, we are alerting on the Application Insights query (exceptions where severityLevel > 1) > 0.
Last night we received 14 alerts of: Error processing event from Partition Id: '#', Owner: '#uuid#', EventHubPath: '#consumer-group#'
I'm having trouble deciphering the error messages- but does this mean that the extension dropped 14 messages?
I'm happy to provide any more information, the most pertenant for this project seems to be the parsedStack:
{
"parsedStack": [
{
"assembly": "System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
"method": "System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
"level": 0,
"line": 0
},
{
"assembly": "System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
"method": "System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
"level": 1,
"line": 0
},
{
"assembly": "System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
"method": "System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
"level": 2,
"line": 0
},
{
"assembly": "Microsoft.Azure.EventHubs, Version=2.1.0.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c",
"method": "Microsoft.Azure.EventHubs.Amqp.AmqpEventHubClient+<CreateConnectionAsync>d__32.MoveNext",
"level": 3,
"line": 0
},
{
"assembly": "System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
"method": "System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
"level": 4,
"line": 0
},
{
"assembly": "System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
"method": "System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
"level": 5,
"line": 0
},
{
"assembly": "System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
"method": "System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
"level": 6,
"line": 0
},
{
"assembly": "Microsoft.Azure.Amqp, Version=2.4.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
"method": "Microsoft.Azure.Amqp.FaultTolerantAmqpObject`1+<OnCreateAsync>d__6.MoveNext",
"level": 7,
"line": 0
},
{
"assembly": "System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
"method": "System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
"level": 8,
"line": 0
},
{
"assembly": "System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
"method": "System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
"level": 9,
"line": 0
},
{
"assembly": "Microsoft.Azure.Amqp, Version=2.4.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
"method": "Microsoft.Azure.Amqp.Singleton`1+<GetOrCreateAsync>d__13.MoveNext",
"level": 10,
"line": 0
},
{
"assembly": "System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
"method": "System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
"level": 11,
"line": 0
},
{
"assembly": "System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
"method": "System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
"level": 12,
"line": 0
},
{
"assembly": "Microsoft.Azure.Amqp, Version=2.4.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
"method": "Microsoft.Azure.Amqp.Singleton`1+<GetOrCreateAsync>d__13.MoveNext",
"level": 13,
"line": 0
},
{
"assembly": "System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
"method": "System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
"level": 14,
"line": 0
},
{
"assembly": "System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
"method": "System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
"level": 15,
"line": 0
},
{
"assembly": "System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
"method": "System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
"level": 16,
"line": 0
},
{
"assembly": "Microsoft.Azure.EventHubs, Version=2.1.0.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c",
"method": "Microsoft.Azure.EventHubs.Amqp.AmqpPartitionReceiver+<CreateLinkAsync>d__15.MoveNext",
"level": 17,
"line": 0
},
{
"assembly": "System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
"method": "System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
"level": 18,
"line": 0
},
{
"assembly": "System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
"method": "System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
"level": 19,
"line": 0
},
{
"assembly": "System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
"method": "System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
"level": 20,
"line": 0
},
{
"assembly": "Microsoft.Azure.Amqp, Version=2.4.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
"method": "Microsoft.Azure.Amqp.FaultTolerantAmqpObject`1+<OnCreateAsync>d__6.MoveNext",
"level": 21,
"line": 0
},
{
"assembly": "System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
"method": "System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
"level": 22,
"line": 0
},
{
"assembly": "System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
"method": "System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
"level": 23,
"line": 0
},
{
"assembly": "Microsoft.Azure.Amqp, Version=2.4.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
"method": "Microsoft.Azure.Amqp.Singleton`1+<GetOrCreateAsync>d__13.MoveNext",
"level": 24,
"line": 0
},
{
"assembly": "System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
"method": "System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
"level": 25,
"line": 0
},
{
"assembly": "System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
"method": "System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
"level": 26,
"line": 0
},
{
"assembly": "Microsoft.Azure.Amqp, Version=2.4.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
"method": "Microsoft.Azure.Amqp.Singleton`1+<GetOrCreateAsync>d__13.MoveNext",
"level": 27,
"line": 0
},
{
"assembly": "System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
"method": "System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
"level": 28,
"line": 0
},
{
"assembly": "System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
"method": "System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
"level": 29,
"line": 0
},
{
"assembly": "Microsoft.Azure.EventHubs, Version=2.1.0.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c",
"method": "Microsoft.Azure.EventHubs.Amqp.AmqpPartitionReceiver+<OnReceiveAsync>d__13.MoveNext",
"level": 30,
"line": 0
},
{
"assembly": "System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
"method": "System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
"level": 31,
"line": 0
},
{
"assembly": "Microsoft.Azure.EventHubs, Version=2.1.0.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c",
"method": "Microsoft.Azure.EventHubs.Amqp.AmqpPartitionReceiver+<OnReceiveAsync>d__13.MoveNext",
"level": 32,
"line": 0
},
{
"assembly": "System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
"method": "System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
"level": 33,
"line": 0
},
{
"assembly": "System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
"method": "System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
"level": 34,
"line": 0
},
{
"assembly": "System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
"method": "System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
"level": 35,
"line": 0
},
{
"assembly": "Microsoft.Azure.EventHubs, Version=2.1.0.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c",
"method": "Microsoft.Azure.EventHubs.PartitionReceiver+<ReceiveAsync>d__31.MoveNext",
"level": 36,
"line": 0
},
{
"assembly": "System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
"method": "System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
"level": 37,
"line": 0
},
{
"assembly": "System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
"method": "System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
"level": 38,
"line": 0
},
{
"assembly": "Microsoft.Azure.EventHubs, Version=2.1.0.0, Culture=neutral, PublicKeyToken=7e34167dcc6d6d8c",
"method": "Microsoft.Azure.EventHubs.Amqp.AmqpPartitionReceiver+<ReceivePumpAsync>d__18.MoveNext",
"level": 39,
"line": 0
}
],
"severityLevel": "Error",
"outerId": "0",
"message": "A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.",
"type": "System.Net.Sockets.SocketException",
"id": "61997365"
}
I defined multiple consumer groups ($Default, production, and staging) in my event hub in order to be able to test my azure functions using the two available slots.
When I statically define the ConsumerGroup in the EventHubTrigger the events trigger the function as expected.
[FunctionName("CardMachineSender")]
public static async Task Run(
[EventHubTrigger("cardmachineeventhub", ConsumerGroup = "staging", Connection = "CardMachineEventHub")]
EventData cardMachineMessage,
ExecutionContext context,
ILogger log)
{But, when I set the ConsumerGroup with a binding expression, to retrieve the value from the application settings, the function is no more triggered.
[FunctionName("CardMachineSender")]
public static async Task Run(
[EventHubTrigger("cardmachineeventhub", ConsumerGroup = "%ConsumerGroupName%", Connection = "CardMachineEventHub")]
EventData cardMachineMessage,
ExecutionContext context,
ILogger log)
{
I currently have a simple azure function app that runs locally and connects to an event hub using a connection string and prints the events to the console. Below is what the .proj dependencies look like:
<PropertyGroup> <TargetFramework>netcoreapp2.2</TargetFramework> <AzureFunctionsVersion>v2</AzureFunctionsVersion> </PropertyGroup> <ItemGroup> <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.EventHubs" Version="3.0.0" /> <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="3.0.3" /> </ItemGroup>
However, when upgrading to Microsoft.Azure.WebJobs.Extensions.EventHubs v4.1.0, the function app seems to start up fine and appears to acquire a host lock lease, but I'm unable to see events. Are there different dependencies needed for v4.1.0?
Trying to understand why the below does not work.
<PropertyGroup> <TargetFramework>netcoreapp2.2</TargetFramework> <AzureFunctionsVersion>v2</AzureFunctionsVersion> </PropertyGroup> <ItemGroup> <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.EventHubs" Version="4.1.0" /> <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="3.0.3" /> </ItemGroup> <ItemGroup>
Create an error loop in a function app by creating an EH triggered function with a non-existent hub name, and let the error loop run for 15 minutes or so. I did this in a Free app in dedicated (not Consumption) without AlwaysOn enabled. Eventually you should start seeing socket errors like the below.
We need to find out what is causing this - it might be a leak in EH client libraries themselves, or perhaps somewhere in the disposal path that we follow when we dispose a ScriptHost instance in error.
I had to restart the function app once in this state to get around this.
2017-02-13T19:11:44.480 Reading host configuration file 'D:\home\site\wwwroot\host.json'
2017-02-13T19:12:06.542 ScriptHost initialization failed
2017-02-13T19:12:06.542 Microsoft.WindowsAzure.Storage.StorageException: Unable to connect to the remote server ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: An attempt was made to access a socket in a way forbidden by its access permissions
at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.Sockets.Socket.InternalBind(EndPoint localEP)
at System.Net.Sockets.Socket.BeginConnectEx(EndPoint remoteEP, Boolean flowContext, AsyncCallback callback, Object state)
at System.Net.Sockets.Socket.UnsafeBeginConnect(EndPoint remoteEP, AsyncCallback callback, Object state)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndGetResponse[T](IAsyncResult getResponseResult)
--- End of inner exception stack trace ---
at Microsoft.WindowsAzure.Storage.Core.Util.StorageAsyncResult`1.End()
at Microsoft.WindowsAzure.Storage.Blob.CloudBlobContainer.EndCreateIfNotExists(IAsyncResult asyncResult)
at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass1`1.<CreateCallback>b__0(IAsyncResult ar)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.WebJobs.Script.BlobLeaseManager.<GetLockBlobAsync>d__32.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.WebJobs.Script.BlobLeaseManager.<GetLockBlobAsync>d__31.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.WebJobs.Script.BlobLeaseManager.<CreateAsync>d__25.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at Microsoft.Azure.WebJobs.Script.ScriptHost.Initialize()
at Microsoft.Azure.WebJobs.Script.ScriptHost.Create(IScriptHostEnvironment environment, ScriptHostConfiguration scriptConfig, ScriptSettingsManager settingsManager)
From @peterhuene on February 22, 2019 3:46
The Azure Functions Event Hubs trigger binding is sending a PartitionContext as part of the trigger metadata, serialized as JSON for extension languages.
However, the type has a public CancellationToken property that should probably be suppressed from JSON serialization.
Here's the JSON object that is serialized for the PartitionContext:
{
"CancellationToken":{
"IsCancellationRequested":false,
"CanBeCanceled":true,
"WaitHandle":{
"Handle":{
"value":2040
},
"SafeWaitHandle":{
"IsInvalid":false,
"IsClosed":false
}
}
},
"ConsumerGroupName":"$Default",
"EventHubPath":"test",
"PartitionId":"0",
"Owner":"40eeeb3a-3491-4072-ba37-59ecdc330b6e",
"RuntimeInformation":{
"PartitionId":"0",
"LastSequenceNumber":0,
"LastEnqueuedTimeUtc":"0001-01-01T00:00:00",
"LastEnqueuedOffset":null,
"RetrievalTime":"0001-01-01T00:00:00"
}
}I'd expect the serialization output to not have the CancellationToken:
{
"ConsumerGroupName":"$Default",
"EventHubPath":"test",
"PartitionId":"0",
"Owner":"40eeeb3a-3491-4072-ba37-59ecdc330b6e",
"RuntimeInformation":{
"PartitionId":"0",
"LastSequenceNumber":0,
"LastEnqueuedTimeUtc":"0001-01-01T00:00:00",
"LastEnqueuedOffset":null,
"RetrievalTime":"0001-01-01T00:00:00"
}
}Copied from original issue: Azure/azure-event-hubs-dotnet#372
Hi,
I've been having this issue for a while now and I cannot figure it out - I'd like to say I've tried everything, but I hope not.
The problem is that the Azure function consistently starves 2 partitions, a lease is taken and then they are just left there until the lease breaks (an hour or something later).
Then two other partitioned are left to be starved.
Whenever I redeploy the function it grabs all of them, but then soon starts to ignore two of them.
It takes about 2 minutes for this problem to start.
I'm running an Azure function (v3 - project config below) that gets data from an event hub with 16 partitions. This is running on a App Service Plan with around 11 nodes (plenty of CPU to spare).
Operating System : Windows
Runtime version : 3.0.13353.0
And I have a deployment slot. This is probably the error as it sometimes doesn't stop consuming the event hub after swapping... but I'm not sure as even if I stop it this problem keeps happening (90% sure, I'll double check). (I have deployed the Slot using an ARM template, maybe something in there thats messing things up? I'm using Terraform for that, and so that is why it looks weird - posted at the bottom)
I've redeployed the function in question, I've messed with every (the 3) setting in the host.json, I've even cycled the machines on the app service plan (set scale to 1 and then to 11 again). I've even tested creating a new consumer group, but the same problem remains.
And this is beyond just upgrading to the latest nuget packages and trying to see if there is anything in my code - lucky me I have a second function that doesn't show this issue to compare with but no luck.
So my guess is that there is something in the extension that's causing this.
Like I said, this is just true for one of two functions who reads the exact same data. But this problem only occurs in significant effect on one of them. The other may have the same problem, but to a much smaller degree for some reason. They both have almost the same setup, output clients for Event Grid and Redis Cache (not provided by Az Function runtime). the only real difference is that the problematic function has an integration with another event hub which is managed by me through an EventHubClient.
Blow are pictures of the the event hub checkpoints as well as what the graph of delay from message enqueued on the event hub looks like.
var lastTimestamp = (DateTime)eventStream.Last().SystemProperties["x-opt-enqueued-time"];
var lastDelay = DateTime.UtcNow - lastTimestamp;
I'm running the following (with some internal packages removed)
<PropertyGroup>
<TargetFramework>netcoreapp3.0</TargetFramework>
<AzureFunctionsVersion>v3</AzureFunctionsVersion>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="GeoCoordinate.NetCore" Version="1.0.0.1" />
<PackageReference Include="GeoLibrary" Version="1.1.0" />
<PackageReference Include="Microsoft.ApplicationInsights" Version="2.14.0" />
<PackageReference Include="Microsoft.Azure.Functions.Extensions" Version="1.0.0" />
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.EventHubs" Version="4.1.1" />
<PackageReference Include="Microsoft.NET.Sdk.Functions" Version="3.0.7" />
<PackageReference Include="TaskTupleAwaiter" Version="1.2.0" />
</ItemGroup>
<ItemGroup>
<None Update="host.json">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
<None Update="local.settings.json">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<CopyToPublishDirectory>Never</CopyToPublishDirectory>
</None>
</ItemGroup>
<ItemGroup>
<Folder Include="Properties\PublishProfiles\" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\..\Service\Service.csproj" />
</ItemGroup>
</Project>
# Run a raw ARM Template to create the slot
resource "azurerm_template_deployment" "function_slot" {
name = "create_function_slot-${azurerm_function_app.function.name}-${var.slotName}-${random_string.random.result}"
parameters = {
"functionSlotName" = "${azurerm_function_app.function.name}/${var.slotName}"
"functionName" = azurerm_function_app.function.name
"slotName" = var.slotName
"appServicePlan_Id" = var.appServicePlanId
"AppSettingsAsJsonAsBase64" = base64encode(jsonencode(merge(local.app_settings,
{
# "FUNCTIONS_WORKER_RUNTIME" = var.function_app_runtime
"FUNCTIONS_EXTENSION_VERSION" = var.function_app_extension_version
# "APPINSIGHTS_INSTRUMENTATIONKEY" = var.applicationInsightsInstrumentationKey
"AzureWebJobsStorage" = var.storageConnectionString
# "WEBSITE_CONTENTAZUREFILECONNECTIONSTRING" = var.storageConnectionString
# "AzureWebJobsDashboard" = var.storageConnectionString
# "WEBSITE_CONTENTSHARE" = "${local.slotName}-content"
}
)))
}
resource_group_name = var.resourceGroupName
deployment_mode = "Incremental"
template_body = <<BODY
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"functionSlotName": {"type": "string", "defaultValue": ""},
"functionName": {"type": "string", "defaultValue": ""},
"slotName": {"type": "string", "defaultValue": ""},
"appServicePlan_Id": {"type": "string", "defaultValue": ""},
"AppSettingsAsJsonAsBase64": {"type": "string", "defaultValue": ""}
},
"variables": {
},
"resources": [
{
"type": "Microsoft.Web/sites/slots",
"apiVersion": "2018-11-01",
"name": "[concat(parameters('functionName'),'/',parameters('slotName'))]",
"location": "[resourceGroup().location]",
"dependsOn": [
],
"kind": "functionapp",
"properties": {
"enabled": true,
"hostNameSslStates": [
],
"serverFarmId": "[parameters('appServicePlan_Id')]",
"reserved": false,
"isXenon": false,
"hyperV": false,
"scmSiteAlsoStopped": false,
"clientAffinityEnabled": true,
"clientCertEnabled": false,
"hostNamesDisabled": false,
"containerSize": 1536,
"dailyMemoryTimeQuota": 0,
"httpsOnly": false,
"redundancyMode": "None",
"siteConfig": {
"alwaysOn": false
}
},
"resources": [
{
"apiVersion": "2018-11-01",
"name": "appsettings",
"type": "config",
"dependsOn": [
"[resourceId('Microsoft.Web/sites/slots', parameters('functionName'),parameters('slotName'))]"
],
"properties": "[base64ToJson(parameters('AppSettingsAsJsonAsBase64'))]"
}
]
}
],
"outputs": {
"functionSlotName": {
"type": "string",
"value": "[parameters('functionSlotName')]"
},
"functionName": {
"type": "string",
"value": "[parameters('functionName')]"
},
"slotName": {
"type": "string",
"value": "[parameters('slotName')]"
}
}
}
BODY
}
Provide the steps required to reproduce the problem
Create an function app to receive event from event hub. Update Microsoft.Azure.Webjobs.Extensions.EventHubs SDK on function app to latest 3.0.6.
In crease the sending speed and send many events to event hub in a very shot time.
Function app should receive the message from event hub normally/
Function failed to receive the message from event hub. There were lots of following 404 not found exceptions:
azure_event_hub.send_event (nhf.main); Action Input: action_type = azure_event_hub.send_event; visitor_id = 01693dd806c30051b9ac6733269003073004c06b007e8; entity = linkvisit; user_properties - HF-SVOC-VISITOR-ID = 016c470d6d0400016be5ce5e168d03073005b06b007e8; user_properties - HF-SVOC-SESSION-ID = 1564560157958; user_properties - HF-SVOC-EVENT-ID = 3575856221913332; user_properties - HF-SVOC-EVENT-VERSION = 1564560168; broker_properties - PARTION_KEY = 1564560157958; pretty_print = unchecked; Action Result Response: Http Response 404
We involved our Azure event hub product group and confirmed that the offset the client provided is not correct and raised such 404 not found issue. Customer confirmed that the SDK Microsoft.Azure.Webjobs.Extensions.EventHubs will store the offset on to storage account.
Customer removed those offsets records on storage account and mitigated this issue.
Provide any related information
Microsoft.Azure.Webjobs.Extensions.EventHubs 3.0.6
https://www.nuget.org/packages/Microsoft.Azure.WebJobs.Extensions.EventHubs/3.0.6
I am currently trying to maximize the number of messages an Azure Function is processing in one "run" using a Event Hub Trigger.
The settings in the host.json file are as follows for this:
"extensions": {
"eventHubs": {
"eventProcessorOptions": {
"maxBatchSize": 64,
"prefetchCount": 128
}
}
},
As is mentioned @lukasvosyka in Azure/azure-functions-host#4480 maxBatchSize is not used at all from the host.json. The reasons for this were among others that he used the Willezone DI libary. We have been doing that as well. Yesterday I replaced the Willezone DI library by the Microsoft default one for DI. I also updated most of the packages to the highest minor versions.
We're using following versions of packages:
We were also using a ConfigurationBuilder in our Startup class and I know this can give issues with the other "configuration" services or containers being overwritten. But I found a workaround for that here: Azure/azure-functions-host#4464 (to be precise - the solution provided by @martinfletcher
(Azure/azure-functions-host#4464 (comment)). In debug mode I can see that multiple containers are loaded.
I put the logging level for the host to trace, and I can see that the valuse for the eventProcessorOptions are set according to the host.json.
2020-01-15T09:11:08.814 [Information] EventHubOptions
{
"BatchCheckpointFrequency": 1,
"EventProcessorOptions": {
"EnableReceiverRuntimeMetric": false,
"InvokeProcessorAfterReceiveTimeout": false,
"MaxBatchSize": 64,
"PrefetchCount": 128,
"ReceiveTimeout": "00:01:00"
},
"PartitionManagerOptions": {
"LeaseDuration": "00:00:30",
"RenewInterval": "00:00:10"
}
}
However - the number of messages received in an array frequently go over 64 debugging with this change locally in Visual Studio. Shouldn't the number of messages in array not go over the set amount because the EventProcessor manages the number of messages that is "sent" to each runtime instance of the function?
Customer reported that TimeoutExceptions from EventProcessorHost is causing too much noise in Application Insights. A sample of the error is as follows:
The operation did not complete within the allocated time 00:01:00 for object session140. For more information on exception types and proper exception handling, please refer to http://go.microsoft.com/fwlink/?LinkId=761101
EventProcessorHost error (Action=Close) : System.TimeoutException: The operation did not complete within the allocated time 00:01:00 for object session140. For more information on exception types and proper exception handling, please refer to http://go.microsoft.com/fwlink/?LinkId=761101
at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.EndClose(IAsyncResult result)
at Microsoft.ServiceBus.Messaging.Amqp.AmqpMessagingFactory.CloseEntityAsyncResult.<>c.b__2_1(CloseEntityAsyncResult thisPtr, IAsyncResult r)
at Microsoft.ServiceBus.Messaging.IteratorAsyncResult1.StepCallback(IAsyncResult result) --- End of stack trace from previous location where exception was thrown --- at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.ServiceBus.Messaging.Amqp.AmqpMessageReceiver.OnEndClose(IAsyncResult result) at Microsoft.ServiceBus.Messaging.ClientEntity.EndClose(IAsyncResult result) at Microsoft.ServiceBus.Messaging.EventHubReceiver.OnEndClose(IAsyncResult result) at Microsoft.ServiceBus.Messaging.ClientEntity.EndClose(IAsyncResult result) at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ServiceBus.Common.TaskHelpers.EndAsyncResult(IAsyncResult asyncResult) at Microsoft.ServiceBus.Messaging.IteratorAsyncResult1.<>c.b__24_1(TIteratorAsyncResult thisPtr, IAsyncResult r)
at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)
System.TimeoutException:
at Microsoft.ServiceBus.Common.AsyncResult.End (Microsoft.ServiceBus, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at Microsoft.ServiceBus.Messaging.Amqp.AmqpObject.EndClose (Microsoft.ServiceBus, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at Microsoft.ServiceBus.Messaging.Amqp.AmqpMessagingFactory+CloseEntityAsyncResult+<>c.b__2_1 (Microsoft.ServiceBus, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at Microsoft.ServiceBus.Messaging.IteratorAsyncResult1.StepCallback (Microsoft.ServiceBus, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35) at Microsoft.ServiceBus.Common.AsyncResult.End (Microsoft.ServiceBus, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35) at Microsoft.ServiceBus.Messaging.Amqp.AmqpMessageReceiver.OnEndClose (Microsoft.ServiceBus, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35) at Microsoft.ServiceBus.Messaging.ClientEntity.EndClose (Microsoft.ServiceBus, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35) at Microsoft.ServiceBus.Messaging.EventHubReceiver.OnEndClose (Microsoft.ServiceBus, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35) at Microsoft.ServiceBus.Messaging.ClientEntity.EndClose (Microsoft.ServiceBus, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35) at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.ServiceBus.Common.TaskHelpers.EndAsyncResult (Microsoft.ServiceBus, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
at Microsoft.ServiceBus.Messaging.IteratorAsyncResult1+<>c.<CallTask>b__24_1 (Microsoft.ServiceBus, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35) at Microsoft.ServiceBus.Messaging.IteratorAsyncResult1.StepCallback (Microsoft.ServiceBus, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
Referencing the link http://go.microsoft.com/fwlink/?LinkId=761101 provided in the error message, for TimeoutExceptions, the suggestion is to incorporate retries, which should already be happening. The log entries showing these exceptions should be transient and the Function should recover and continue execution. These logs should be logged as warning instead of errors.
The nuget page (https://www.nuget.org/packages/Microsoft.Azure.WebJobs.Extensions.EventHubs) still links to the old github repo (azure-webjobs-sdk) under "Source Repository".
We need to add a log when EventHub output sends the messages to help CRI investigations.
https://github.com/Azure/azure-functions-eventhubs-extension/blob/dev/src/Microsoft.Azure.WebJobs.Extensions.EventHubs/Triggers/EventHubAsyncCollector.cs#L99
Users today cannot control when their Event Hub starts to run from - it generally starts from the very beginning which can mean lots of data which adds latency/wasted compute/etc. Users should be able to specify how far back the Event Hubs trigger starts.
From: https://feedback.azure.com/forums/355860/suggestions/19085314
The current EventHubListener.cs implementation progresses the checkpoint if there is an unhandled exception in user code.
See EventHubListener.cs:
Unfortunately, internal exceptions in DryIoC are treated as user exceptions, and the user has no way of catching them.
Azure/azure-webjobs-sdk#2432
Azure/azure-functions-host#5240
Fixing the DryIoC exceptions is one thing. There is however no guarantee that such situations will not occur again.
I propose a slighty modified flow, where the user is able to decide whether the checkpoint should be progressed or not, if there is an unhandled user exception. This can be done through a setting in host.json.
I forked the repo and made a pull request to demonstrate. There is also a MyGet package if you want to test. Currently, in the PR, it is hardcoded to not progress the checkpoint.
PR: xzuttz#1
If you are interested in pursuing the proposal I am happy to complete the pull request with any further instructions.
Albeit I am not able to really reproduce the DryIoC exception in a test environment, we can reproduce that DryIoC exceptions, in the function scope, will be treated as user exceptions, by doing the following:
The HTTP trigger will not return.
The logs will say that the DryIoC exception came from the HTTP trigger context, thus the checkpoint will be progressed by the EventHubListener.
I believe the maximum event message size is wrong in the binding. It was probably not updated when Event Hub upgraded to 1MB limit of messages: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-quotas
This should be 1 MB (or close to it, depending on how much space you want to leave for headers)
(I'm still finding my way around Azure, so apologise if this is not the approrpiate place to raise this issue).
We have a Node.js Function App configured with an Azure Event Hubs trigger.
Because this is a fairly critical piece of infrastructure to us, we are alerting on the Application Insights query (exceptions where severityLevel > 1) > 0.
Approximately 5 times a day we receive a log for a Microsoft.Azure.EventHubs.Processor.LeaseLostException<Microsoft.WindowsAzure.Storage.StorageException> "The lease ID specified did not match the lease ID for the blob."
This logs with a severityLevel of 1 (which I understand to be 'information'). From what I can see, there's nothing I can do about these, and no action I can take, and they're just a fact of Azure life (as discussed in issue #5).
However these errors periodically (and seemingly randomy) get re-thrown as System.AggregateException "One or more errors occurred. (The lease ID specified did not match the lease ID for the blob.)". Which has a severityLevel of 3 (error).
Is there anything we can do to silence these- or are there actions we should be taking because of them? I don't want to get people into the habit of ignoring alarms, but at the moment that's my best advice.
Getting this periodically from the EventHubListener. Any help is appreciated.
Time 4:56:17 PM
Exception type System.AggregateException
Exception message Unable to cast object of type 'System.Collections.Generic.List1[System.Diagnostics.Activity]' to type 'System.Collections.Generic.IEnumerable1[System.Diagnostics.Activity]'.
LogLevel Error
Category Host.Triggers.EventHub
HostInstanceId 41890ee6-adff-43c7-aec1-25745d35eb7d
System.AggregateException: An error occurred while writing to logger(s). (Unable to cast object of type 'System.Collections.Generic.List1[System.Diagnostics.Activity]' to type 'System.Collections.Generic.IEnumerable1[System.Diagnostics.Activity]'.) ---> System.InvalidCastException: Unable to cast object of type 'System.Collections.Generic.List1[System.Diagnostics.Activity]' to type 'System.Collections.Generic.IEnumerable1[System.Diagnostics.Activity]'.
at System.Collections.Generic.DictionaryExtensions.GetValueOrDefault[T](IDictionary2 dictionary, String key) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Shared\DictionaryExtensions.cs:line 13 at Microsoft.Azure.WebJobs.Logging.ApplicationInsights.ApplicationInsightsLogger.StartTelemetryIfFunctionInvocation(IDictionary2 stateValues) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Logging.ApplicationInsights\ApplicationInsightsLogger.cs:line 468
at Microsoft.Azure.WebJobs.Logging.ApplicationInsights.ApplicationInsightsLogger.BeginScope[TState](TState state) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Logging.ApplicationInsights\ApplicationInsightsLogger.cs:line 435
at Microsoft.Extensions.Logging.Logger.BeginScope[TState](TState state)
--- End of inner exception stack trace ---
at Microsoft.Extensions.Logging.Logger.BeginScope[TState](TState state)
at Microsoft.Extensions.Logging.LoggerExtensions.BeginFunctionScope(ILogger logger, IFunctionInstance functionInstance, Guid hostInstanceId) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Loggers\Logger\LoggerExtensions.cs:line 64
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsyncCore(IFunctionInstanceEx functionInstance, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 110
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(IFunctionInstance instance, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 87
at Microsoft.Azure.WebJobs.Host.Executors.TriggeredFunctionExecutor1.TryExecuteAsync(TriggeredFunctionData input, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\TriggeredFunctionExecutor.cs:line 56 at Microsoft.Azure.WebJobs.EventHubs.EventHubListener.EventProcessor.ProcessEventsAsync(PartitionContext context, IEnumerable1 messages)
at Microsoft.Azure.EventHubs.Processor.PartitionPump.ProcessEventsAsync(IEnumerable1 events) ---> (Inner Exception #0) System.InvalidCastException: Unable to cast object of type 'System.Collections.Generic.List1[System.Diagnostics.Activity]' to type 'System.Collections.Generic.IEnumerable1[System.Diagnostics.Activity]'. at System.Collections.Generic.DictionaryExtensions.GetValueOrDefault[T](IDictionary2 dictionary, String key) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Shared\DictionaryExtensions.cs:line 13
at Microsoft.Azure.WebJobs.Logging.ApplicationInsights.ApplicationInsightsLogger.StartTelemetryIfFunctionInvocation(IDictionary`2 stateValues) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Logging.ApplicationInsights\ApplicationInsightsLogger.cs:line 468
at Microsoft.Azure.WebJobs.Logging.ApplicationInsights.ApplicationInsightsLogger.BeginScope[TState](TState state) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Logging.ApplicationInsights\ApplicationInsightsLogger.cs:line 435
at Microsoft.Extensions.Logging.Logger.BeginScope[TState](TState state)<---
Please add support for an interval based check-pointing option to the EventHub trigger/extension of WebJobs. Currently all that is supported is a per-batch or per-N-batches check-pointing option.
In high throughput scenarios it is a very useful feature to be able to set a frequency of checkpoints based on a time interval as opposed to a batch count. This allows you to have predictable checkpoints at regular intervals, which also allows you to reliably predict your storage needs for check-pointing.
This allows you to take the storage latency out of band of the event processing pipeline entirely, which in extremely high throughput scenarios is very valuable.
When defining an Event Hub trigger you can specify a time interval to make batches at.
Not available. Only per-batch or per-N-batches.
Calculate your message throughput and determine roughly the number of batches that will occur in the time interval you desire batches to be made at, and use that is your per-N-batches option setting. This is an imprecise workaround.
For the .NET framework version of the event processor host I had developed such an interval based check-pointing algorithm. It would be nice to see something similar to this supported in web jobs/functions.
We need better instrumentation of what is happening inside the event hubs extension (similar to the logging improvements recently made for blobs). What I'd like to see captured:
When "cardinality": "many" is set on an Event Hubs trigger (as it should be for best perf), the error below is thrown unless the original input is valid json. This conflicts with v1 behavior and breaks the template example
System.Private.CoreLib: Exception while executing function: Functions.eventhub. Microsoft.Azure.WebJobs.Host: Exception binding parameter 'myEventHubMessages'. Microsoft.Azure.WebJobs.Host: Binding parameters to complex objects (such as 'Object') uses Json.NET serialization.
1. Bind the parameter type as 'string' instead of 'Object' to get the raw values and avoid JSON deserialization, or
2. Change the queue payload to be valid json. The JSON parser failed: Unexpected character encountered while parsing value: E. Path '', line 0, position 0.
Workaround:
If you know your input may not be valid json, add "dataType": "string" to trigger binding.
Example function.json:
{
"bindings": [
{
"type": "eventHubTrigger",
"name": "myEventHubMessages",
"direction": "in",
"eventHubName": "samples-workitems",
"connection": "EventHubReceiver",
"cardinality": "many",
"dataType": "string"
}
]
}
=> System.Collections.Generic.Dictionary`2[System.String,System.Object] Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: Functions.eventhub ---> System.InvalidOperationException: Exception binding parameter 'myEventHubMessages' ---> System.InvalidOperationException: Binding parameters to complex objects (such as 'Object') uses Json.NET serialization. 1. Bind the parameter type as 'string' instead of 'Object' to get the raw values and avoid JSON deserialization, or 2. Change the queue payload to be valid json. The JSON parser failed: Unexpected character encountered while parsing value: T. Path '', line 0, position 0.
at Microsoft.Azure.WebJobs.Host.Triggers.PocoTriggerArgumentBinding2.ConvertAsync(TMessage value, Dictionary2 bindingData, ValueBindingContext context) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Triggers\TriggerArgumentBinding\PocoTriggerArgumentBinding.cs:line 57
at Microsoft.Azure.WebJobs.Host.ArrayTriggerArgumentBinding2.BindAsync(TTriggerValue value, ValueBindingContext context) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Triggers\TriggerArgumentBinding\ArraryTriggerArgumentBinding.cs:line 37 at Microsoft.Azure.WebJobs.Host.Triggers.TriggeredFunctionBinding1.BindCoreAsync(ValueBindingContext context, Object value, IDictionary`2 parameters) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Triggers\TriggeredFunctionBinding.cs:line 57
--- End of inner exception stack trace ---
at Microsoft.Azure.WebJobs.Host.Executors.DelayedException.Throw() in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\DelayedException.cs:line 27
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstance instance, ParameterHelper parameterHelper, ILogger logger, CancellationTokenSource functionCancellationTokenSource) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 460
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstance instance, ParameterHelper parameterHelper, IFunctionOutputDefinition outputDefinition, ILogger logger, CancellationTokenSource functionCancellationTokenSource) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 426
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstance instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 231
--- End of inner exception stack trace ---
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstance instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 275
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(IFunctionInstance functionInstance, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 75
fail: Function.eventhub[0]
Additional context:
PR with first pass at fix: Azure/azure-functions-host#3170
Issue was brought to attention here: Azure/azure-functions-host#2815
Hi,
I have a continuous web job that reads random data from a CSV file and send them to Azure Event hub.
When the job is started, it starts to send data to event hub instantly and is supposed to continue until it is manually stopped.
The issue is that at a certain point of time, the web job stops sending data to event but would eventually restart sending data automatically after some time.
Here is a snapshot of the Event Hub metric for the past 7 days:
I enabled logging and here is the exception thrown:
<EventData>
<Data>Application: DataGenerator.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.UnauthorizedAccessException
at System.IO.__Error.WinIOError(Int32, System.String)
at System.IO.FileStream.Init(System.String, System.IO.FileMode, System.IO.FileAccess, Int32, Boolean, System.IO.FileShare, Int32, System.IO.FileOptions, SECURITY_ATTRIBUTES, System.String, Boolean, Boolean, Boolean)
at System.IO.FileStream..ctor(System.String, System.IO.FileMode, System.IO.FileAccess, System.IO.FileShare, Int32, System.IO.FileOptions, System.String, Boolean, Boolean, Boolean)
at System.IO.StreamWriter.CreateFile(System.String, Boolean, Boolean)
at System.IO.StreamWriter..ctor(System.String, Boolean, System.Text.Encoding, Int32, Boolean)
at System.IO.StreamWriter..ctor(System.String, Boolean)
at DataGenerator.Program.SendData()
at DynamicClass.lambda_method(System.Runtime.CompilerServices.Closure, DataGenerator.Program, System.Object[])
at Microsoft.Azure.WebJobs.Host.Executors.VoidMethodInvoker`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].InvokeAsync(System.__Canon, System.Object[])
at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`1+<InvokeAsync>d__0[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithWatchersAsync>d__31.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__2c.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(System.Threading.Tasks.Task)
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__13.MoveNext()
Exception Info: Microsoft.Azure.WebJobs.Host.FunctionInvocationException
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__13.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<TryExecuteAsync>d__1.MoveNext()
at Microsoft.Azure.WebJobs.Host.Executors.ExceptionDispatchInfoDelayedException.Throw()
at Microsoft.Azure.WebJobs.JobHost+<CallAsyncCore>d__7.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
at Microsoft.Azure.WebJobs.JobHost.Call(System.Reflection.MethodInfo)
at DataGenerator.Program.Main()
</Data>
Can anyone please help?
Thanks
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google โค๏ธ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.