Keyvault roles (Key Vault Secrets Officer, Key Vault Secrets User) are not in preview anymore, it needs to be updated in raft_deploy.py

When using the command line and specifying an alternate defaults.json file with the --defaults-context-path parameter the raft.py looks for the defaults.json file co-located with the raft.py file.

This causes a problem if you have multiple deployments and interact with the via an alias.

Decouple authentication definitions from the code
Potential options

1. Move full authentication definition into Keyvault. This way agent must deserialize the full secret value
2. Keep Authentication type definition as part of the task, and only keep the actual authentication information in the keyvault

Agent: 0-fuzz-1 Tool: RESTler State: Error
Details:
error : Input string '64.5' is not a valid integer. Path 'total_unique_test_cases', line 10, position 35.
Agent: 1-fuzz-2 Tool: RESTler State: Error
Details:
error : Input string '81.0' is not a valid integer. Path 'total_unique_test_cases', line 10, position 35.

Access to a deployment's API

The proposed feature is to add support to restrict access to the deployment based on membership to an AAD group. This would allow fine grain control over who has access to the API and removes the possibility of someone with AAD guest access being able to submit jobs, etc.

This change would affect who would have permissions to authenticate with the service. If you did not belong to the AAD group, your authentication to the service should fail.

Storage Data

Current access to storage accounts is currently controlled with Azure RBAC which already supports AAD groups. There will be no change to this method of access control.

if endpoint has "http" and useSSL is not set, then assume useSSL is false

if endpoint has "https" and useSSL is not set, then assume useSSL is true

Remove

        if len(test_target_container_names) == 0:
            # no bridge needed, since there are not "services under test" deployed
            # and therefore we are testing something deployed externally
            bridge_name = None

from raft_local.py because your system under test might be running directly on your host, not inside a docker image (own or RAFT). Setting bridge_name from 'host' to None disables localhost for docker.

One option

1. copy /ado/variables/version-variables.yml to CLI folder
2. when running raft.py --version <- parse version variables out from the yaml, and print that out

Currently webhook can sent a sample data to the registered endpoint.

As an improvement when implementing webhook handling - add option to send the sample data back to the user in the reply. This will improve user experience when creating Logic Apps for handling webhooks triggered by RAFT

Currently if we have a deployment named "demo" we will create a service bus named "demo-raft-servicebus" this name must be unique across azure. If a demo service has been created somewhere else deployment will fail.

We should add the same uniqueness we've added elsewhere using the digits from the subscription as part of the name.

I am getting the below error when trying to deploy RAFT:

Creating Key Vault event subscription
args: az eventgrid event-subscription create --name OnSecretChanged --source-resource-id /subscriptions/XXXXX/resourceGroups/XX-raft/providers/Microsoft.KeyVault/vaults/XX-raft-1fcb-kv --endpoint /subscriptions/XXXXX/resourceGroups/XX-raft/providers/Microsoft.Web/sites/XX-raft-1fcb-orchestrator/functions/OnSecretChanged --endpoint-type azurefunction --included-event-types Microsoft.KeyVault.SecretNewVersionCreated

std error: ERROR: Deployment failed. Correlation ID: XXXXX. Destination azure endpoint not found. Resource details: resourceId: /subscriptions/XXXXX/resourceGroups/XX-raft/providers/Microsoft.Web/sites/XX-raft-1fcb-orchestrator/functions/OnSecretChanged. Resource should pre-exist before attempting this operation. Activity id:XXXX, timestamp: 3/25/2021 12:47:45 AM (UTC).

AZ Version: 2.20.0

Allow docker deployment of a service under test as part of the RAFT job.

ACIs support access to each other over localhost:port.

The only addition is needed is to add schema for specifying service under test

When I use the CompilerConfig members (UseBodyExamples, UseQueryExamples, DiscoverExamples, ExamplesDirectory), they all produce a "Could not find on object of type 'CompileConfiguration'" in the RESTler agent. Other members, like "CustomDictionary" work fine so is this because the Examples parameters have not been implemented in Agent yet? Is there a way to work around this limitation.

When looking at the apiservice swagger, we should see the swagger files that tools defined in their schema.json file integrated.
jobdefinition-> tasks->rafttask -> toolConfiguration ->

Perf improvement:
If all tools in a job run send Completed event, then consider the job to be completed

The default scope string is currently defined as [|sprintf "%s/.default" auth.client|]
It should be [|"/.default"|]

See https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-permissions-and-consent

• Use RESTler BVT
• Add ZAP as part of the BVT run

For ZAP tool API scan is failing to import TopicClient and Message from azure.servicebus. These have been replaced with ServiceBusClient and ServiceBusMessage respectively. Affects scan.py

I am having a .NET core application and we have several microservices. The swagger APIs are authenticated with tokens. I wanted to implement fuzzing using RESTler but I am unable to figure out how to integrate RESTler with an API that requires authentication. Can someone provide a working example please.

Right now PR to support YAML job definition only for Python CLI (since Python code converts YAML to JSON and then send that to the service)

Does it make sense for service to support both YAML and JSON (or only keep JSON, and have YAML to JSON conversion in a CLI) ?

Seeing this error during deployment.

Assigning Key Vault roles
adding Read permission
granting permissions
args: az keyvault secret set --description "v312 Service Principal authentication credentials" --file /mnt/d/released-cli/v3.1/raft_sdk/.tmp/sp.json --name RaftServicePrincipal --vault-name v312-raft-6e2b-kv
std error: ERROR: Caller is not authorized to perform action on resource.
If role assignments, deny assignments or role definitions were changed recently, please observe propagation time.
Caller: appid=04b07795-8ddb-461a-bbee-02f9e1bf7b46;oid=78cd9cd1-7446-4a3e-85b2-63375f97deaa;iss=https://sts.windows.net/72f988bf-86f1-41af-91ab-2d7cd011db47/
Action: 'Microsoft.KeyVault/vaults/secrets/setSecret/action'
Resource: '/subscriptions/da3e8787-6e2b-4cae-b201-6e4cd71be189/resourcegroups/v312-raft/providers/microsoft.keyvault/vaults/v312-raft-6e2b-kv/secrets/raftserviceprincipal'
Assignment: (not found)
Vault: v312-raft-6e2b-kv;location=centralus

Even though we are waiting for the service principal to be visible in AAD there still seems to be a timing issue,
Re-running the deploy step always works.

We are pulling the version from the wrong DLL when reporting metrics from the orchestrator.

In the discussion of Step 3: Choose Configuration Options, it is said that four of the config options are required, yet the table of options shows six as required ("Yes" in the table's "Required?" column)

Running in a local deployment on the new version with agent-utilities.
I encounter the following error with multiple tools.
My configuration template:

{
  "testTasks" : {
    "targetConfiguration": {
      "apiSpecifications": [
        "<specification-url>"
      ],
      "endpoint": "<endpoint-url>"
    },
    "tasks": [
      {
        "toolName": "ZAP",
        "outputFolder": "zap",
        "keyVaultSecrets": [ "TOKEN" ],
        "authenticationMethod": {
            "Token": "Token"
        }
      }
    ]
  }
}

Where Token is file in \cli\local\secrets\ containing:
Basic <base64-auth-token>=

Execution fails in container before running any tasks.

zap2docker container log:

Traceback (most recent call last):
File "run.py", line 18, in <module>
token = raft.auth_token()
File "/raft-tools/tools/ZAP/../../libs/python3/raft.py", line 76, in auth_token
raise Exception(response.text)
Exception: {"error":"Traceback (most recent call last): File \"/raft-tools/agent-utilities/auth/python3/token/token.py\", line 13, in <module> token = token_from_env_variable(sys.argv[1]) File \"/raft-tools/agent-utilities/auth/python3/token/token.py\", line 10, in token_from_env_variable raise Exception(f\"Token environment variable is not set {env_variable_name}\")Exception: Token environment variable is not set RAFT_TOKEN"}

Replacing toolName in config with Dredd will give similar error.
dredd container log:

Authentication Method: Token
Error: Request to http://localhost:8085/auth/Token/Token failed with status code 400
at ClientRequest.<anonymous> (/raft-tools/libs/node-js/raft.js:179:34)
at Object.onceWrapper (events.js:483:26)
at ClientRequest.emit (events.js:376:20)
at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:647:27)
at HTTPParser.parserOnHeadersComplete (_http_common.js:126:17)
at Socket.socketOnData (_http_client.js:515:22)
at Socket.emit (events.js:376:20)
at addChunk (internal/streams/readable.js:309:12)
at readableAddChunk (internal/streams/readable.js:284:9)
at Socket.Readable.push (internal/streams/readable.js:223:10)
(node:8) UnhandledPromiseRejectionWarning: SyntaxError: Unexpected end of JSON input
at JSON.parse (<anonymous>)
at IncomingMessage.<anonymous> (/raft-tools/libs/node-js/raft.js:81:57)
at IncomingMessage.emit (events.js:388:22)
at endReadableNT (internal/streams/readable.js:1336:12)
at processTicksAndRejections (internal/process/task_queues.js:82:21)

Inspecting the containers with Docker Hub shows the environment variable RAFT_TOKEN is being set correctly: Basic <base64-auth-token>=.

Ignoring the Token with the following "ugly" work-around seems to solve.
Replace the last 3 lines in token.py with:

if __name__ == "__main__":
    print("Basic <base64-auth-token>=")

I'm guessing the issue occurs when reading the environment variable:

auth_params = os.environ.get(env_variable_name)

Running RESTler in a local deployment with the following configuration:

{
  "testTasks" : {
    "targetConfiguration": {
      "apiSpecifications": [
        "<specification-url>"
      ],
      "endpoint": "<endpoint-url>"
    },
    "tasks": [
      {
        "toolName": "RESTler",
        "outputFolder": "restler",
        "keyVaultSecrets": [ "Token" ],
        "authenticationMethod": {
            "TxtToken": "Token"
        },
        "toolConfiguration": {
          "tasks": [
            {
              "task": "Compile",
              "compileConfiguration": {
                "MutationsSeed": 12098347
              }
            },
            {
              "task": "Test",
              "runConfiguration": {
                "Duration": "00:10:00",
                "useSsl": false
              }
            },
            {
              "task": "TestFuzzLean",
              "runConfiguration": {
                "Duration": "00:10:00",
                "useSsl": false
              }
            },
            {
              "task": "Fuzz",
              "runConfiguration": {
                "Duration": "00:10:00",
                "useSsl": false
              }
            }
          ]
        }
      }
    ]
  }
}

Execution fails after "Compile" task, because of timeout from garbage collection.

Docker container log:

Raft.Agent failed due to: System.Exception: RESTler engine failed. See logs in /work_dir_80ff13a4-bec3-427f-8ef3-1ad1d2aea399 directory for more information.

at Microsoft.FSharp.Core.PrintfModule.PrintFormatToStringThenFail@1433.Invoke(String message) in D:\workspace\_work\1\s\src\fsharp\FSharp.Core\printf.fs:line 1433

at [email protected](ProcessResult _arg1) in /home/vsts/work/1/s/src/Agent/RESTlerAgent/RESTlerDriver.fs:line 256

at Microsoft.FSharp.Control.AsyncPrimitives.CallThenInvokeNoHijackCheck[a,b](AsyncActivation`1 ctxt, FSharpFunc`2 userCode, b result1) in D:\workspace\_work\1\s\src\fsharp\FSharp.Core\async.fs:line 404

at Raft.RESTlerDriver.RESTlerInternal.startProcessAsync@150-37.Invoke(AsyncActivation`1 ctxt) in /home/vsts/work/1/s/src/Agent/RESTlerAgent/RESTlerDriver.fs:line 150

at [email protected](AsyncActivation`1 ctxt) in D:\workspace\_work\1\s\src\fsharp\FSharp.Core\async.fs:line 577

at [email protected](AsyncActivation`1 ctxt) in D:\workspace\_work\1\s\src\fsharp\FSharp.Core\async.fs:line 577

at [email protected](AsyncActivation`1 ctxt) in D:\workspace\_work\1\s\src\fsharp\FSharp.Core\async.fs:line 577

at Raft.RESTlerDriver.RESTlerInternal.startProcessAsync@134-23.Invoke(AsyncActivation`1 ctxt) in /home/vsts/work/1/s/src/Agent/RESTlerAgent/RESTlerDriver.fs:line 134

stdout-RESTlerEngine.txt log file:

Initializing: Garbage collection every 30 seconds.
Terminating garbage collection. Waiting for max 300 seconds.
'Exceeded Timeout'

While running raft_local with ZAP I noticed this in the docker logs.

WARNING: The directory '/home/zap/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
zapcli 0.10.0 requires python-owasp-zap-v2.4==0.0.14, but you have python-owasp-zap-v2-4 0.0.18 which is incompatible.
zapcli 0.10.0 requires six==1.10.0, but you have six 1.15.0 which is incompatible.

RESTler compile step task supports compilation of a list of swagger files (more of an advanced use case)
(ZAP task meanwhile supports only one file)

RAFT - currently only supports one swagger file definition per task.

Figure out schema changes that will allow "clean" definition of multiple swagger definitions

Currently in Orchestartor we are waiting for conatiner group completion before marking job as completed. Seems like Azure is having some sync issues. Several times my container finish executing, meanwhile Azure is still displaying container state as "waiting" and container group state as "creating".

If all containers within group terminated then it is a sufficient condition to assume that the job run is completed

Is it possible to use raft_local.py to fuzz-test purely locally, without cloud service?

All /cli/samples have a run Python script that expects a default.json and aborts since I don't have it. I was able to create one with default empty strings for all attributes using raft.py job create, but the attributes seem to be for Azure configuration. So I guess I do not need default.json when using raft_local.py!?

Is there a sample or description somewhere on how to use raft_local.py? Do I need jobStatusWebhookUrl and bugFoundWebhookUrl for raft_local.py, too?

In detail I tried the following using raft_local.py's help output and some guesswork:

1. change storage, secrets = init_local(work_dir) to storage, secrets, events_sink = init_local(work_dir) in raft_local.py
2. python raft_local.py local init
3. Define a job1.json as in step 3 of https://github.com/microsoft/rest-api-fuzz-testing/blob/main/docs/how-to-submit-a-job.md, changing apiSpecification to a list of strings, thus making it valid json
4. python raft_local.py job create --file job1.json
5. The log shows:
   creating job ...
   pulling Zapp and Dredd from Docker
   finally the traceback (most recent call last):
   File "raft_local.py", line 654, in
   run(vars(args))
   File "raft_local.py", line 594, in run
   cli.new_job(job_config, args.get('jobStatusWebhookUrl'), args.get('bugFoundWebhookUrl'))
   File "raft_local.py", line 523, in new_job
   test_task_container_names = self.start_test_tasks(job_config, task_index, test_services_startup_delay, job_id, work_dir, job_dir, job_events, bridge_name)
   File "raft_local.py", line 392, in start_test_tasks
   tt['targetConfiguration'] = testTasks['targetConfiguration']
   File "/home/davef/git/rest-api-fuzz-testing/cli/raft_sdk/raft_common.py", line 52, in getitem
   return super(RaftJsonDict, self).getitem(key)
   KeyError: 'targetConfiguration'
6. Since I am most interested in RESTler, I changed job1.json to job2.json containing only one task, with toolName being RESTler.
7. python raft_local.py job create --file job2.json
8. as 5)
9. jobStatusWebhookUrl and bugFoundWebhookUrl sound optional for raft_local.py, so (why) do I need them?
   How do I get my task.config to contain targetConfiguration (see function start_test_tasks)? I cannot find targetConfiguration in RaftJobConfig or in https://github.com/microsoft/rest-api-fuzz-testing/blob/main/docs/how-to-submit-a-job.md.
   Is it generated from targetEndpointConfiguration? My job2.json contains:

{
  "testTasks": {
    "targetEndpointConfiguration": {
      "apiSpecification": [
          "file:///home/davef/git/mediform/scheduling-server/openapi_tmp.json"
        ],
      "endpoint": "http://localhost:8088"
    },
    "tasks": [
      {
        "toolName": "RESTler",
        "outputFolder" : "restler-results-output-folder"
      }
    ]
  }
}

Is this related to point 3 above? Or is my openapi_tmp.json problematic? I created it from my openapi.yaml using a generic yaml2json transformation.

Hi,
could you provide a sample that shows how to use RESTler inline and external examples via RAFT?

• Description of new samples
• Descriptions of new target options, which should include your observation that the PostRun command if only used to copy logs can be ignored if the server is able to be redirected to write the logs to the share.

We use a self-signed certificate during development and get this when running Dredd:

Error:
Error: unable to verify the first certificate
Stacktrace:
Error: unable to verify the first certificate
at TLSSocket.onConnectSecure (_tls_wrap.js:1497:34)
at TLSSocket.emit (events.js:315:20)
at TLSSocket._finishInit (_tls_wrap.js:932:8)
at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:706:12)

is there a way to work around this either via RAFT or Dredd itself?

Almost all test tools support disabling of certificate validation.

Expose this as an option in a job config to allow users to turn that off.

If HTTPs endpoint requires custom cert - add ability to add the cert before running test tools

    "Id": "c8aabe2f-0fe5-4ee5-a894-d1cdd2a2852a",
    "EventType": "BugFound",
    "Subject": "BugFound",
    "Data": {
      "Tool": "RESTler",
      "JobId": "petstore3-test-fuzz-lean-249a6935-ba6d-4aa5-bc81-4f267874fe9a",
      "AgentName": "0-test-fuzz-lean",
      "Metadata": {
        "Some": null
      },

Execution of Schemathesis (in local deployment) fails to import package 'six' in Docker container.
Log shows following:

Installing collected packages: six, oauthlib, requests-oauthlib, isodate, uamqp, msrest, azure-core, azure-common, azure-servicebus, applicationinsights
  Attempting uninstall: six
    Found existing installation: six 1.16.0
    Uninstalling six-1.16.0:
      Successfully uninstalled six-1.16.0
Successfully installed applicationinsights-0.11.10 azure-common-1.1.27 azure-core-1.13.0 azure-servicebus-7.0.1 isodate-0.6.0 msrest-0.6.21 oauthlib-3.1.0 requests-oauthlib-1.3.0 six-1.15.0 uamqp-1.4.0

Although six-1.15.0 is "Successfully installed", the run.py script fails with:

Traceback (most recent call last):
  File "/usr/local/bin/schemathesis", line 5, in <module>
    from schemathesis.cli import schemathesis
  File "/usr/local/lib/python3.9/site-packages/schemathesis/__init__.py", line 1, in <module>
    from . import fixups, hooks, runner, serializers, targets
  File "/usr/local/lib/python3.9/site-packages/schemathesis/runner/__init__.py", line 7, in <module>
    from ..checks import DEFAULT_CHECKS
  File "/usr/local/lib/python3.9/site-packages/schemathesis/checks.py", line 3, in <module>
    from .exceptions import get_status_code_error
  File "/usr/local/lib/python3.9/site-packages/schemathesis/exceptions.py", line 8, in <module>
    from jsonschema import ValidationError
  File "/usr/local/lib/python3.9/site-packages/jsonschema/__init__.py", line 22, in <module>
    from jsonschema.validators import (
  File "/usr/local/lib/python3.9/site-packages/jsonschema/validators.py", line 11, in <module>
    from six import add_metaclass
ModuleNotFoundError: No module named 'six'

Installation of one of the packages is causing the issue.
A possible work-around is adding: --ignore-installed six as an argument when installing packages.
In \cli\raft-tools\tools\Schemathesis\run.py, line 14:

subprocess.check_call([sys.executable, "-m", "pip", "install", "--no-cache-dir", "--root", "/tmp", "-r", os.path.join(raft_libs_dir, "requirements.txt"), "--ignore-installed", "six"])

Maintaining the "classic" release pipeline in ADO is proving to be a problem. Changes made to the yaml pipelines have to be duplicated in the release pipeline.

There are however advantages to the release pipelines. It keeps a nice record of the releases and it's easy to have approval gates.
One of the problems in doing all this in yaml is that yaml at this time will only support 1 global variable group. Because we use variable groups for the staging environment as well as the production environments this becomes a problem.

The solution is to use the release pipeline to simply run the yaml pipelines.

Running Schemathesis in a local deployment with the following configuration:

{
  "testTasks" : {
    "targetConfiguration": {
      "apiSpecifications": [
        "<specification-url>"
      ],
      "endpoint": "<endpoint-url>"
    },
    "tasks": [
      {
        "toolName": "Schemathesis",
        "outputFolder": "schemathesis",
        "keyVaultSecrets": [ "<token>" ],
        "authenticationMethod": {
            "TxtToken": "<token>"
        }
      }
    ]
  }
}

Where <token> is file in \cli\local\secrets\ containing:
Basic <base64-auth-token>=
Authentication token is handled correctly in run.py for Dredd and ZAP but not for Schemathesis.
Results in following if-statement skipped:

if token:
  args.extend(["-H", f"Authorization: {token}"])

Hard-coding the same token seems to work:
In \cli\raft-tools\tools\Schemathesis\run.py, after line 41:

args.extend(["-H", f"Authorization: Basic <base64-auth-token>="])

P.S.:
Consider adding an argument to disable TLS verification, I'm currently using the following:

args.extend(["--request-tls-verify", "false"])

Initially we were loading secrets keyvault on every job create (that is not very efficient, since keyvault is very slow and does not like lots of operations). Now we cache secrets in orchestartor on start (this requires orchestrator restart when secrets change)

We can do the following to refresh secret cache when required:

• Add Service Bus QUEUE: reload secrets
• Add reload command that can be triggered from CLI
• Hook that up into Azure KeyVault events, so they post on the queue when keyvault has updates

Minimize service knowledge of the BugFound and JobStatus contents, and treat those events as JSON blobs

During deployment we are assigning the event grid event before the orchestrator is running. The command requires that the orchestrator is running to succeed.

Move the assignment to the end of the deployment after the services are started to ensure they are running.

I've setup my RAFT config with:

    "toolName": "RESTler",
    "outputFolder": "restler-logs",
    "toolConfiguration": {
      "tasks": [
        {
          "task": "compile",
      "compileConfiguration": 
	{
		"discoverExamples": true,
    "usebodyexamples": true,
    "UseQueryExamples": true,
    "ExamplesDirectory": "."

and my API json has:

    "parameters": [
      {
        "name": "id",
        "in": "path",
        "description": "Id of the object",
        "required": true,
        "schema": {
          "type": "string",
          "format": "guid"
        },
          "examples": 
        {
            "example": {
              "$ref": "./examples.json"
            }
        }

So I would expect the Examples.json to get copied to the work dir, but I get

Unhandled exception. System.AggregateException: One or more errors occurred. (Could not resolve the JSON path '/work_dir_6ac5804a-4632-4ec4-aa45-a0dd090ecfde/examples.json' with the full JSON path '/work_dir_6ac5804a-4632-4ec4-aa45-a0dd090ecfde/examples.json'.)
---> System.InvalidOperationException: Could not resolve the JSON path '/work_dir_6ac5804a-4632-4ec4-aa45-a0dd090ecfde/examples.json' with the full JSON path '/work_dir_6ac5804a-4632-4ec4-aa45-a0dd090ecfde/examples.json'.
---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
---> System.IO.FileNotFoundException: Could not find file '/work_dir_6ac5804a-4632-4ec4-aa45-a0dd090ecfde/examples.json'.
File name: '/work_dir_6ac5804a-4632-4ec4-aa45-a0dd090ecfde/examples.json'

from the RESTler compiler and the examples.json is not in the workdir

Can RAFT be configured to use basic authentication (user/pass)?

System.IO.IOException: Interrupted system call
at System.IO.Enumeration.FileSystemEnumerator1.FindNextEntry(Byte* entryBufferPtr, Int32 bufferLength) at System.IO.Enumeration.FileSystemEnumerator1.MoveNext()
at Microsoft.FSharp.Collections.SeqModule.IsEmpty[T](IEnumerable1 source) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\seq.fs:line 702 at Raft.RESTlerDriver.RESTlerInternal.getRunExperimentFolder(String fuzzingWorkingDirectory, DateTime runStartTime) in /home/vsts/work/1/s/src/Agent/RESTlerAgent/RESTlerDriver.fs:line 167 at [email protected](Boolean _arg1) in /home/vsts/work/1/s/src/Agent/RESTlerAgent/RESTlerDriver.fs:line 471 at Microsoft.FSharp.Control.AsyncPrimitives.CallThenInvokeNoHijackCheck[a,b](AsyncActivation1 ctxt, FSharpFunc2 userCode, b result1) in F:\workspace\_work\1\s\src\fsharp\FSharp.Core\async.fs:line 417 at Microsoft.FSharp.Control.Trampoline.Execute(FSharpFunc2 firstAction) in F:\workspace_work\1\s\src\fsharp\FSharp.Core\async.fs:line 109