azure / azure-functions-durable-powershell Goto Github PK
View Code? Open in Web Editor NEWPowerShell SDK for writing Durable Functions apps
License: MIT License
PowerShell SDK for writing Durable Functions apps
License: MIT License
Currently, the DurableTask.Core.History.HistoryEvent
is accessed from the global namespace due to a naming conflict. Ideally, a shorter type name would be used, and there would be a cleaner way to access it than through the global namespace.
Currently, cmdlets in the external durable SDK have External
and E
suffixes and corresponding print statements to distinguish them from cmdlets present in the PowerShell worker. These should be removed before the SDK is fully released.
This issue is still happening in Azure Durable Functions via HTTP starter function.
Apparently this has been fixed, but it does not actually seem to be, see:
To rule out any issues in my own code & functions, I also tested using Microsoft's own examples, showing the issue persists:
Using template:
The orchestrator and activity are running just fine.
The Starter works fine until it gets to the following line:
$Status = Get-DurableStatus -InstanceId $InstanceId
Starter returns:
2024-05-03T18:35:58Z [Information] INFORMATION: DurableClient started
2024-05-03T18:35:58Z [Information] INFORMATION: Started orchestration with ID = 'b937fc9d-a34d-4407-a876-c9cdd2f18c2e', FunctionName = 'regAPOrch'
2024-05-03T18:35:58Z [Error] EXCEPTION: Response status code does not indicate success: 401 (Unauthorized).
Orchestrator returns:
2024-05-03T18:39:51Z [Information] INFORMATION: DurableOrchestrator: started. Input: Hello
2024-05-03T18:39:51Z [Information] INFORMATION: DurableOrchestrator: finished.
2024-05-03T18:39:51Z [Information] OUTPUT: Hello Tokyo
2024-05-03T18:39:51Z [Information] <id>: Function 'regAPOrch (Orchestrator)' completed.
Activity returns:
2024-05-03T18:39:41Z [Information] INFORMATION: DurableActivity(Tokyo) started
2024-05-03T18:39:51Z [Information] INFORMATION: DurableActivity(Tokyo) finished
2024-05-03T18:39:51Z [Information] Executed 'Functions.regAPAct'
I have a function that uses Invoke-WebRequest in order to pull data from an external API. Here's a sanitzed version of the function:
function Get-APIResource {
param(
[Parameter (Mandatory=$true)]$Resource,
[Parameter (Mandatory=$true)]$SearchQuery
)
$BaseURL = ""
$headers = @{
ApiCode = $env:APISecret
UserName = $env:PIUser
Secret = $env:APIKey
'Content-Type' = 'application/json'
}
$ResourceURL = $BaseURL + $Resource + '/query?search=' + $SearchQuery
$i = 1
try {
do {
Write-Warning "Getting page $i of $Resource"
$Response = Invoke-WebRequest -Method GET -Uri $ResourceURL -Headers $headers -TimeoutSec 30 -ErrorAction Stop
$Response = $Response.Content | ConvertFrom-Json
If ($Response.items){
Write-Warning "($i) Retrieved $($Response.items.count) items"
$Response.items | ForEach-Object{
$_
#$ReturnList.Add($_)
}
}
If ($Response.item){
Write-Warning "($i) Retrieved $($Response.item.count) ITEM"
$Response.item | ForEach-Object{
$_
#$ReturnList.Add($_)
}
}
$ResourceURL = $Response.pagedetails.nextPageUrl
If ($null -eq $ResourceURL){
Write-Warning "($i) ResourceURL is null"
} Else {
Write-Warning "($i) Next page URL: $ResourceURL"
}
$i++
} while ($null -ne $ResourceURL)
}
catch {
Write-Error "Connecting to API Failed."
throw "API Error: $($_.Exception.Message)"
}
}
I'm using it in an Activity function, like so:
param($Context)
# Set DataType
$DataType = 'Products'
# Initialize default return object
$returnObject = [PSCustomObject]@{
$DataType = "Problem accessing data."
}
# Get data
try{
$filter = @{} | ConvertTo-JSON
$Data = Get-APIResource -Resource Products -SearchQuery $filter
}
catch{
Write-Warning "Error retrieving $DataType."
Write-Error "ERROR: $($Error[0].Exception.Message)"
If ($Error[0].InnerException){
Write-Error "INNER ERROR: $($Error[0].Exception.InnerException)"
}
throw $returnObject
}
If (-not $Data){
Write-Warning "`$Data variable is blank. Failed to get $DataType."
return $returnObject
}
# Return
[PSCustomObject]@{
$DataType = $Data
} | ConvertTo-Json -Depth 100
And this is being called by an Orchestrator function:
param($Context)
Write-Output "Starting Orchestrator."
$RetryOptions = New-DurableRetryOptions -FirstRetryInterval (New-TimeSpan -Seconds 5) -MaxNumberOfAttempts 3
Write-Output "Retrieving Config Items from Autotask"
$APIData = Invoke-DurableActivity -FunctionName "Func-RetrieveAPIData" -RetryOptions $RetryOptions
# Build object (add metadata) to send to data lake
$SendObject = @{
APIData = $APIData
sourceName = "API"
}
# Convert to JSON to prep for sending
$SendJson = $SendObject | ConvertTo-Json -Depth 100
# Send to the data lake function
Write-Output "Sending Data to Data Lake function"
Invoke-DurableActivity -FunctionName "Func-SendtoDataLake" -Input $SendJson
When I look at Log Stream in the Azure Function, I can see the progress of Invoke-WebRequest
. When it runs normally, it looks like this:
PROGRESS: Reading response stream... (Number of bytes read: 0)
PROGRESS: Reading response stream... (Number of bytes read: 3276)
PROGRESS: Reading response stream... (Number of bytes read: 13276)
PROGRESS: Reading response stream... (Number of bytes read: 23276)
PROGRESS: Reading response stream... (Number of bytes read: 27864)
...
...
PROGRESS: Reading response stream... (Number of bytes read: 256662)
PROGRESS: Reading web response completed. (Number of bytes read: 256662)
However, when it has a problem, it will halt after a random number of bytes read. Verbose logs will continue to output from normal operation of the Azure Function. The Activity Function times out after 10 minutes, at which point the Orchestrator finishes.
Normally, the API request takes under 30 seconds to complete -- usually it completes in under 10 seconds. But since the web stream from Invoke-WebRequest
is hanging the Activity Function up, I cannot implement Durable Retry or Durable Wait in the Orchestrator function. I've tried wrapping each Invoke-WebRequest
in Start-ThreadJob
within Get-APIResource
and using Wait-Job
with a 30 second timeout. I'm able to properly throw an error from the API function to the Activity function, and the Orchestrator function catches it, but the Activity Function still sits there and waits until 10 minutes to time out because the thread job is locked up thanks to the Invoke-WebRequest
stream.
I cannot reproduce this problem locally (via manual execution of Get-APIResource
or via Azure Function debugging in VS Code) or on an Azure VM. I can fairly reliably reproduce this problem within Azure Automation Accounts / Runbooks and Azure Functions.
I've been chasing this error down since at least September 2023. I'm happy to share any other required information to help track this down.
The external SDK needs to be imported manually by the user in their profile.ps1
. We will need to update our templates to demonstrate how to do this.
We will need to modify the templates for VSCode, and core-tools. There may be more template sources in other places as well that will need updating.
Currently, Microsoft.Azure.WebJobs.Extensions.DurableTask.100.7.0.nupkg
must be present in the test/E2E/durableApp
folder for the test app to work properly. This should be removed (as well as the corresponding entry in the extensions.csproj
file) before the SDK is fully released.
Follow up to: #53 (comment)
We should keep our JSON serialization strategy compatible with the worker. Otherwise, we risk serialization errors. Creating this ticket to track that dependency.
Currently, pipeline runs on Linux may fail due to either orchestrator timeouts or due to HttpClient
timeouts. These may be due to worker concurrency settings not being optimized for test runs, but investigation is required to make runs more reliable.
WaitAny
writes the first DurableTask
to complete to the output pipe. However, the information displayed is not useful, showing type information. It would be good to make sure the output at least matches what is present in the PowerShell worker or displays more useful information (for example, the external event name in a ExternalEvent
task).
Currently, the only E2E test included is OrchestratorReplaysCurrentUtcDatetime
. Other tests present in the PowerShell worker must be added before the external SDK is ready to be released.
Related item https://github.com/Azure/azure-functions-powershell-worker/issues/748
tl;dr: When using Fan out/fan in pattern, and one or more activity fails, the final output in the orchestrator function is dropped.
Still not able to handle activity errors in an orchestrator function when using the fan/out fan/in pattern. Although it seems to report the failure there is no way to get a working output for combinations of successful and failed activities at all. It either discards the successful output, or ignores the failure. Using same code to above issue but using the new SDK.
Trigger (Service Bus)
param([string] $SbMsg, $TriggerMetadata)
$ErrorActionPreference = "Stop"
<#
Service bus message received:
{
"TestFilter": {
"version": "1.0.0",
"environment": "testEnv",
"itemName": [
"test1",
"test2",
"test3",
"test4",
"test5"
],
"appComponent": "testcomponent",
"application": "testApp"
}
}
#>
# Declare 5 cities to be randomly picked later
$city = @("London", "Paris", "New York", "Tokyo", "Sydney")
# Create a hashtable the orchestrator could use to iterate over
foreach ($item in $TriggerMetadata.TestFilter.itemName) {
$hashtableout += @{
$item = @{
city = $city | Get-Random
itemName = $item
}
}
}
# Start the orchestration
$InstanceId = Start-DurableOrchestration -FunctionName "of-test" -Input $hashtableout -ErrorAction Stop
Write-Host "Started orchestration with ID = '$InstanceId'"
Activity Function
param($name)
$ErrorActionPreference = 'Stop'
$actStopwatch = [System.Diagnostics.Stopwatch]::StartNew()
if ($($name.itemName) -eq "test3") { throw }
elseif ($($name.itemName) -eq "test1") { Start-Sleep -Seconds 3}
elseif ($($name.itemName) -eq "test5") { Start-Sleep -Seconds 8 }
elseif ($($name.itemName) -eq "test4") { Start-Sleep -Seconds 50 }
# Finalise outputs
$actStopwatch.stop()
[decimal]$elapsedSecods = $actStopwatch.Elapsed.TotalMinutes
$actTotalTime = "{0:N2}" -f $elapsedSecods
Write-Information -MessageData "[$($name.itemName)] Completed in [$actTotalTime]..."
$name.add('runResults', "Deployment Succeeded. Runtime [$actTotalTime]")
"[$($name.itemName)] Completed in [$actTotalTime]"
So when using the below Orchestrator code everything works as expected regarding the error handling and getting final output from the orchestrator function. The orchestrator schedules the tasks and the final output includes any successful activity outputs as well as failures caught and handled within the orchestrator function. The problem with this orchestrator pattern however is the tasks are asynchronous and therefore do not run efficiently in that they queue behind each other (using sleep statements in the activity highlights this as can be seen below in the scheduled times)
Orchestrator (Working error handling, but pattern not optimal)
using namespace System.Net
param($Context)
$ErrorActionPreference = 'Stop'
$output = @()
foreach ($property in $Context.Input.psobject.properties.name) {
# invoke the activity function
try {
$output += Invoke-DurableActivity -FunctionName "af-test" -Input $Context.Input.$property
}
catch {
$output += "[$($property)] Failed"
}
}
return $output
So the final output highlights the failure meaning the catch block works as expected BUT the scheduling of activities is done in series (function chaining) and therefore causes a completion time build up (last activity might run quickest but has to wait for all others to complete first)
Changing to the Fan out/fan in pattern, the behaviour changes:
using namespace System.Net
param($Context)
$ErrorActionPreference = 'Stop'
$output = @()
$ParallelTasks =
foreach ($property in $Context.Input.psobject.properties.name) {
# invoke the activity function
try {
Invoke-DurableActivity -FunctionName "af-test" -Input $Context.Input.$property -NoWait
}
catch {
$output += "[$($property)] Failed"
}
}
$output += Wait-DurableTask -Task $ParallelTasks
return $output
Changing to $ErrorActionPreference = 'Continue' allows the orchestrator to complete, but there is no output returned:
Moving the try/catch works partially in that it seems to capture the failures, but disregards the successful runs from the output:
using namespace System.Net
param($Context)
$ErrorActionPreference = 'Stop'
$output = @()
$ParallelTasks =
foreach ($property in $Context.Input.psobject.properties.name) {
# invoke the activity function
Invoke-DurableActivity -FunctionName "af-test" -Input $Context.Input.$property -NoWait
}
try {
$output += Wait-DurableTask -Task $ParallelTasks
}
catch {
$output += "[$($property)] Failed"
}
return $output
It also does not collect multiple failures:
Using the same logic as in the fan out/fan in section here: https://github.com/Azure/azure-functions-durable-powershell/blob/main/test/E2E/durableApp/DurablePatternsOrchestrator/run.ps1 .
using namespace System.Net
param($Context)
$ErrorActionPreference = 'Continue'
$output = @()
$ParallelTasks = @()
foreach ($property in $Context.Input.psobject.properties.name) {
# invoke the activity function
try {
$ParallelTasks += Invoke-DurableActivity -FunctionName "af-test" -Input $Context.Input.$property -NoWait
}
catch {
$ParallelTasks += "[$($property)] Failed"
}
}
$output += Wait-DurableTask -Task $ParallelTasks
return $output
Again runs successfully, but does not handle the error or produce any final output. However, using this same orchestrator code but with no failing activities works and produces the final output:
So, to me it seems that whenever one or more activities fail, it just nukes any final output or captures only one.
I've tried relentlessly but cannot get the Fan out/fan in orchestrator to properly output when activities fail. I've also tried setting various different options for the $ErrorActionPreference to no avail. Would be great if error handling could work similar to when the chaining pattern is used and final output is generated with success or failures or both. Or perhaps there is a way to individually query the task results when using the Fan out/fan in pattern to achieve a similar result?
An owner or administrator of this repository has previously indicated that this repository can not be migrate to GitHub inside Microsoft because it is going public, open source, or it is used to collaborate with external parties (customers, partners, suppliers, etc.).
π βοΈ In order to keep Microsoft secure, we require repository owners and administrators to review this repository and regularly renew the intent whether to opt-in or opt-out of migration to GitHub inside Microsoft which is specifically intended for private or internal projects.
βOnly users with admin
permission in the repository are allowed to respond. Failure to provide a response will result to your repository getting automatically archived. π
If this repository can not be migrated to GitHub inside Microsoft, you can opt-out of migration by replying with a comment on this issue containing one of the following optout
command options below.
@gimsvc optout --reason <staging|collaboration|delete|other>
Example:
@gimsvc optout --reason staging
Options:
staging
: My project will ship as Open Sourcecollaboration
: Used for external or 3rd party collaboration with customers, partners, suppliers, etc.delete
: This repository will be deleted because it is no longer needed.other
: Other reasons not specified
If the circumstances of this repository has changed and you decide that you need to migrate, then you can specify the optin
command below. For example, the repository is no longer going public, open source or require external collaboration.
@gimsvc optin --date <target_migration_date in mm-dd-yyyy format>
Example:
@gimsvc optin --date 03-15-2023
@gimsvc optin --date <target_migration_date>
When opting-in to migrate your repository, the
--date
option is required followed by your specified migration date using the format:mm-dd-yyyy
@gimsvc optin --date 03-15-2023
Opt-out
@gimsvc optout --reason <staging|collaboration|delete|other>
When opting-out of migration, you need to specify the
--reason
.
staging
- My project will ship as Open Source
collaboration
- Used for external or 3rd party collaboration with customers, partners, suppliers, etc.
delete
- This repository will be deleted because it is no longer needed.
other
- Other reasons not specified
Examples:
@gimsvc optout --reason staging
@gimsvc optout --reason collaboration
@gimsvc optout --reason delete
@gimsvc optout --reason other
Related Issue: Azure/azure-functions-durable-extension#1922
If I build up a ps object with sub-objects in an orchestration function and send that to an activity function, it ends up as a hashtable on the other side:
$InputData = [PSCustomObject]@{
Data = $Data
Documents = $Documents
SyncDataType = $SyncDataType
}
Invoke-DurableActivity -FunctionName 'Sync-DataWithDocuments' -Input $InputData
Maybe this is just a limitation but I want to be sure before I use workarounds. I can get it to work if I do the following:
$InputData = [PSCustomObject]@{
Data = $Data
Documents = $Documents
SyncDataType = $SyncDataType
}
Invoke-DurableActivity -FunctionName 'Sync-DataWithDocuments' -Input ($InputData | ConvertTo-Json -Depth 100)
Then in the activity function
function Sync-DataWithDocuments {
Param($Object)
$Object = $Object | ConvertTo-Json -Depth 100 | ConvertFrom-Json
}
In order to protect and secure Microsoft, private
or internal
repositories in GitHub for Open Source which are not related to open source projects or require collaboration with 3rd parties (customer, partners, etc.) must be migrated to GitHub inside Microsoft a.k.a GitHub Enterprise Cloud with Enterprise Managed User (GHEC EMU).
βοΈ Please RSVP to opt-in or opt-out of the migration to GitHub inside Microsoft.
βOnly users with admin
permission in the repository are allowed to respond. Failure to provide a response will result to your repository getting automatically archived.π
Reply with a comment on this issue containing one of the following optin
or optout
command options below.
β Opt-in to migrate
@gimsvc optin --date <target_migration_date in mm-dd-yyyy format>
Example:
@gimsvc optin --date 03-15-2023
OR
β Opt-out of migration
@gimsvc optout --reason <staging|collaboration|delete|other>
Example:
@gimsvc optout --reason staging
Options:
staging
: This repository will ship as Open Source or gopublic
collaboration
: Used for external or 3rd party collaboration with customers, partners, suppliers, etc.delete
: This repository will be deleted because it is no longer needed.other
: Other reasons not specified
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.