Currently, if the request to Dev Ops for the build status fails, we failed the whole task. We should add retry logic and fail after 3 retries. For sample failed task, see https://azfunc.visualstudio.com/Azure%20Functions/_releaseProgress?releaseId=3371&_a=release-environment-logs&environmentId=6733#.

4d

Update PowerShell E2E tests to run in the context of Integration Tests (using the latest prerelease version of the Core Tools)

cc @safihamid

Tracking item to figure out how to automate ColdStart JIT Trace as part of nightly builds

1d

1 day

cc @Francisco-Gamino

This work item depends on setting up windows private stamp.

@brettsam - when you get a chance can you please add details here for metadata generator integration tests? Is this a good repo to track adding tests for this?

If this is not the right place, please close this issue.

Enumerate exact steps needed to update both the Rapid Update feed and the host bits in the associated storage account

time estimation: 3 days

1. Create a pipeline at each worker repro (example below focuses on the PowerShell)

   • Build worker, skip tests, push nuget package to our internal DevOps Artifacts feed

2. Create a pipeline in the azure-function-integration-tests that checks out the Host dev branch, updates the references of the packages for the workers (created in 1), and builds the Host bits using the build-extensions.ps1 script.

3. Create a pipeline in the Core Tools or azure-function-integration-tests repo to build the Core Tools

   • Update package reference in the Core Tools to use the latest created packages for the workers and Host
   • Build Core Tool v3, mark this version as ‘PreRelease`, push/copy to storage account so that this version can be access via our tooling feed
   • Push core tools to a storage account
   • run e2e tests pillung core tools generated above

2d

Here is an example of an issue that should be detected early or prevented: Azure/azure-functions-host#6545

Schedule Host pipeline to build for integration tests
Push the Functions zip package to blob storage

1d

Update existing Host pipeline to update the references of the packages for the PowerShell worker, build Host bits using the build-extensions.ps1 script. Push nuget package to our internal DevOps Artifacts feed

Automate those steps with a script

time estimation: 5 days

Create new Azure DevOps pipeline to build and execute the PowerShell E2E tests. These will need to run in the context of Integration Tests using the latest prerelease version of the Core Tools.

Functions metrics tests

• Identify the metrics that we care about
• Create functions apps in our test region
• Leverage self-alerts service to validate the metrics. If the expected result are not present file a sev 3 icm

For the August demo, set test function app in Consumption and Premium and look at function executions. Send an email notification instead of ICM

Research orchestrator options for the Integration Tests build.

Possible options are:

1. Azure DevOps release pipeline
2. Durable function

Create a pipeline in the PowerShell worker repo, build worker, skip tests, push nuget package to our internal DevOps Artifacts feed

Research private stamp testing using the latest Host and worker bits from the nightly build.

We will need these details for the test plan review which will take place at the end of August 2020.

Here is some additional details from the workitem (https://github.com/pragnagopa/azure-functions-integration-tests/issues/1) that @pragnagopa created on her repo.
Please reach out to her in case you have any questions on this area.

Here is a potential flow to get started, but again, I could be missing important details.

1. Pull Functions Host bits from CI dev nightly build (we can get these from our internal prerelease feed)
2. Prepare Host bits for private stamp (there might be an additional step between 1 and 2
3. Upload to blob via Update RU feed for Windows
4. Pull latest RU feed
5. Update RapidUpdateFeed.xml and upload host.zip to the blob storage used by the private stamp
6. Kick off test runs on each language worker - with FunctionAppURLs on private stamp

Once you have this information, please add it to the test doc.

Once the private stamp is set up:

• enumerate steps needed to run the tests themselves on the private stamp

• answer the following question: How to schedule tests? How to run the tests? Where to see the test results?

• Investigate both Windows and Linux scenarios

time estimation: 5 days

cc @safihamid

Tracking item to figure out how to automate ColdStart JIT Trace as part of nightly builds

The commit id is returned with the build information after the pipeline is started.

Research options for a reporting system. This will depend on the type of orchestrator option.

Possible options to look into are DevOps Releases or a Durable function.

After Azure/azure-functions-powershell-worker#521 is fixed, add an E2E test validating this

Create a pipeline in the Core Tools or azure-function-integration-tests repo to build the Core Tools

• Update package reference in the Core Tools to use the latest created packages for the workers and Host from the internal DevOps * Artifacts feed. Build Core Tool v3, mark this version as ‘PreRelease`
• Push core tools to a storage account
• Run e2e tests pillung core tools generated above

• Need to automate: pulling down RU feed and uploading host bits to storage account.

Create a function app which will:

1. Copy files from prod RU feed into custom RU feed.
2. Get host bits from storage account, and use UpdateStorageSiteExtensions.exe tool to upload the bits to the custom RU feed.

time estimation: 5d