Describe the bug

Verbose logging within the runspace operations doesn't seem to be writing to stream 4.

Steps to reproduce

1. Execute Invoke-AzOpsRepository with -Verbose flag
2. Logging will display until it reaches the ForEach-Object -Parallel

To ensure logging is consistently handled within the module, existing functions need to be updated to use the custom Write-AzOpsLog function.

Add all AzOpsEnvVariables in action.yml

Describe the bug

System branch is sometimes fallen behind master

Steps to reproduce

Screenshots

Describe the bug
The implementation guide link on the README page is returning a 404

Steps to reproduce

1. Click on the link

Screenshots

Describe the solution you'd like

Describe the solution you'd like

#19

Policy assignment for built-in policy fails with pipeline when assigned in the portal.

Steps to reproduce:

1. Assign a built-in Deny Policy (e.g. "Network interfaces should not have public IPs")
2. Discover environment (note: "Identity": null, json property is created)
3. Create a branch and change policy (e.g Description)
4. Create PR -> Push check will fail with the following error

VERBOSE: 8:42:35 PM - Checking deployment status in 5 seconds
Write-AzOpsLog: /action/entrypoint.ps1:93
Line |
  93 |              Write-AzOpsLog -Level Error -Topic "git" -Message $PSItem …
     |              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | [2020-06-22 20:42:41.5546] (git) 8:42:41 PM - The deployment
     | 'Microsoft.Management-managementGroups_Test' failed with
     | error(s). Showing 2 out of 2 error(s). Status Message: Unable
     | to process template language expressions for resource
     | '/providers/Microsoft.Management/managementGroups/Test/providers/Microsoft.Authorization/roleAssignments/5069d15e-f99a-5491-8aed-1f4baa9e27e1' at line '156' and column '9'. 'The language expression property 'identity' doesn't exist, available properties are 'apiVersion, location, sku, properties, deploymentResourceLineInfo, scope, resourceId, referenceApiVersion, condition, isConditionTrue, isTemplateResource, isAction, provisioningOperation'.' (Code:InvalidTemplate)  Status Message: The resource operation completed with terminal provisioning state 'Failed'. (Code: ResourceDeploymentFailure)  - At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details. (Code: DeploymentFailed)    - ***   "status": "Failed",   "error": ***     "code": "ResourceDeploymentFailure",     "message": "The resource operation completed with terminal provisioning state 'Failed'.",     "details": [       ***         "code": "DeploymentFailed",         "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.",         "details": [           ***             "code": "BadRequest",             "message": "***\r\n  \"error\": ***\r\n    \"code\": \"InvalidTemplate\",\r\n    \"message\": \"Unable to process template language expressions for resource '/providers/Microsoft.Management/managementGroups/Test/providers/Microsoft.Authorization/roleAssignments/5069d15e-f99a-5491-8aed-1f4baa9e27e1' at line '156' and column '9'. 'The language expression property 'identity' doesn't exist, available properties are 'apiVersion, location, sku, properties, deploymentResourceLineInfo, scope, resourceId, referenceApiVersion, condition, isConditionTrue, isTemplateResource, isAction, provisioningOperation'.'\",\r\n    \"additionalInfo\": [\r\n      ***\r\n        \"type\": \"TemplateViolation\",\r\n        \"info\": ***\r\n          \"lineNumber\": 156,\r\n          \"linePosition\": 9,\r\n          \"path\": \"\"\r\n        ***\r\n      ***\r\n    ]\r\n  ***\r\n***"           ***         ]       ***     ]   *** *** (Code:Conflict)    CorrelationId: 9dc7c3ac-4cba-45aa-ad01-037f0b1fd41c

AzOps get in a bit of a mess when the above happens.

The vwan RG is created in NorthEurope, then the vwan is deployed in westeurope.

However later the same deployment, it fails as it tried to re-create the RG. For reference, AZOPS_DEFAULT_DEPLOYMENT_LOCATION=northeurope

Error code

{
  "code": "DeploymentFailed",
  "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.",
  "details": [
    {
      "code": "Conflict",
      "message": "{\r\n  \"error\": {\r\n    \"code\": \"InvalidResourceGroupLocation\",\r\n    \"message\": \"Invalid resource group location 'northeurope'. The Resource group already exists in location 'westeurope'.\"\r\n  }\r\n}"
    }
  ]
}

Steps to reproduce

1. Deploy new ES foundation to westeurope, add subscription ID and enable resource deployment
2. Run the pull pipeline to sync changes
3. Assign deploy VWAN policy to the management MG (containing the MSDN sub), with the below assignment

{
  "Location": "northeurope",
  "Name": "Deploy-VWAN",
  "ResourceType": "Microsoft.Authorization/policyAssignments",
  "Properties": {
    "Scope": "/providers/Microsoft.Management/managementGroups/ES-management",
    "NotScopes": [],
    "DisplayName": "Deploy-vWAN",
    "Description": "",
    "PolicyDefinitionId": "/providers/Microsoft.Management/managementGroups/ES/providers/Microsoft.Authorization/policyDefinitions/Deploy-vWAN",
    "Parameters": {
      "vwanname": {
        "value": "logicore-vwan"
      },
      "vwanRegion": {
        "value": "westeurope"
      },
      "rgName": {
        "value": "logicore-global-vwan"
      }
    }
  },
  "Identity": {
    "type": "SystemAssigned"
  }
}

Add support for validating that git config --system core.longpaths true is set.

Explore migrating from Azure Pipelines to GitHub Actions now that fork triggers are supported on Pull Requests.

https://docs.github.com/en/actions/reference/events-that-trigger-workflows#pull_request_target

Support Discovery in environment with duplicate display name for management group and subscription

It's not immediately clear what format is expected for the AZURE_CREDENTIALS parameter that is passed in during docker run (e.g via an Azure DevOps Pipelines variable).

My tenant has only the "Tenant Root Group" and two subscriptions assigned to it. No child managmement goups have been created.

1. Start discovery using the GH action.
2. Following discovery error occurs:

Write-AzOpsLog: /action/entrypoint.ps1:93
Line |
  93 |              Write-AzOpsLog -Level Error -Topic "git" -Message $PSItem …
     |              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | [2020-06-22 14:26:32.3805] (git) [2020-06-22 14:26:32.3721]
     | (pwsh) Cannot find any Management Groups. Does the Service
     | Principal/User have the appropriate privileges on the root
     | Management Group or is the Management Group hierarchy not yet
     | created?

Workaround:
Creating a single management group under Tenant Rout Group solves this issue.

Design approach for injecting schema attribute into Az parameter files to assist with template development.

Describe the solution you'd like

Describe the bug

Within the Pull model we currently depend upon the default branch being defined as 'main', this will cause us challenges if repository are using alternative names such as 'master' / 'trunk' etc.

We should refactor the models to inject the default branch name at runtime from the supported SCM's.

Describe the bug

Need to investigate the Docker image sizes and why certain layers have increased exponentially

As per: https://docs.microsoft.com/en-us/powershell/module/az.accounts/Connect-AzAccount?view=azps-4.2.0

The below:

Should Be:

Connect-AzAccount -Tenant $credentials.tenantId -ServicePrincipal -Credential $credential -SubscriptionId $credentials.subscriptionId -WarningAction SilentlyContinue | Out-Null

as it gives errors:

Write the Azure 'correlation id' to AzOps logs for any ARM interaction failures to support root cause investigation.

Rename default branch from master to main

• Parameterizing the deployment location for DINE is on hold (work item added for policy eng team)
• Policy refresh; new services supporting private endpoints
• Policy refresh to create alerts on key platform events
• Enhance bootstrapping experience (do complete mgmt sub; security, logs etc.)
• Enhance UIDefinition
• Update azopsreference with roleDefinition
• Update docs, samples etc on how to do roleDefinition/roleAssignment
• Update docs to explain when subsequent roleAssignment is needed for DINE policies when crossing scopes (and how to do it)

Add PSScriptAnalyzer to Pipeline to improve test coverage and code quality.

Support dynamic discovery of partial mg root based on RBAC permissions if AZOPS_SUPPORT_PARTIAL_MG_DISCOVERY is set to
(without specifying AZOPS_PARTIAL_MG_DISCOVERY_ROOT)

Describe the bug
2020-08-24T08:13:40.4035157Z DEBUG: AzureQoSEvent: CommandName - Invoke-AzRestMethod; IsSuccess - True; Duration - 00:00:00.0894976;
2020-08-24T08:13:40.9152134Z DEBUG: Finish sending metric.
2020-08-24T08:13:40.9153210Z DEBUG: 8:13:40 AM - InvokeAzRestMethodCommand end processing.
2020-08-24T08:13:41.0609669Z �[91mWrite-AzOpsLog: �[0m/var/lib/app/entrypoint.ps1:88
2020-08-24T08:13:41.0610177Z �[96mLine |
2020-08-24T08:13:41.0610741Z �[96m 88 | �[0m �[96mWrite-AzOpsLog -Level Error -Topic "entrypoint" -Message �[0m …
2020-08-24T08:13:41.0611313Z �[96m | �[91m ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2020-08-24T08:13:41.0611892Z �[91m�[96m | �[91m[08:13:40.9345] (entrypoint) The property 'count' cannot be
2020-08-24T08:13:41.0612426Z �[96m | �[91mfound on this object. Verify that the property exists.
2020-08-24T08:13:41.0612870Z �[0m
2020-08-24T08:13:41.2795276Z Post job cleanup.

Steps to reproduce

1. Run AzOps-Pull (manually or scheduled)

Screenshots

• Add initial guide (.github/CONTRIBUTING.md)
• Add git workflow process
• Add branch strategy

Add sleep task (15 seconds~) to Azure Pipelines.

This is to allow the Scope job time to complete before In-a-Box initiates a deletion process.

The current AZURE_CREDENTIALS secret format includes the following fields which are not needed:

• displayName
• name
• subscriptionId

Whilst these could optionally be specified, they seem to just add unnecessary complexity when generating the AZURE_CREDENTIALS secret value.

This feature request is to remove the dependency on these values in Azure/AzOps and to update documentation in Azure/Enterprise-Scale.

Describe the bug
2020-08-25T06:10:13.1257994Z DEBUG: [06:10:13.1250] (ConvertTo-AzOpsState) Statepath is azops/Tenant Root Group (abc)/.AzState/Microsoft.Authorization_roleAssignments-36d6bb24-e1e2-4a58-aeef-49f67c045865.parameters.json
2020-08-25T06:10:13.1262966Z DEBUG: [06:10:13.1255] (ConvertTo-AzOpsState) Initiating function ConvertTo-AzOpsState process
2020-08-25T06:10:13.1273913Z VERBOSE: [06:10:13.1266] (ConvertTo-AzOpsState) AzOpsState file not found. Creating new: azops/Tenant Root Group (abc)/.AzState/Microsoft.Authorization_roleAssignments-36d6bb24-e1e2-4a58-aeef-49f67c045865.parameters.json
2020-08-25T06:10:13.1288036Z VERBOSE: [06:10:13.1280] (ConvertTo-AzOpsState) Exporting AzOpsState to azops/Tenant Root Group (abc)/.AzState/Microsoft.Authorization_roleAssignments-36d6bb24-e1e2-4a58-aeef-49f67c045865.parameters.json
2020-08-25T06:10:13.1290944Z VERBOSE: [06:10:13.1286] (ConvertTo-AzOpsState) Ordering object
2020-08-25T06:10:13.2572251Z �[91mWrite-AzOpsLog: �[0m/var/lib/app/entrypoint.ps1:105
2020-08-25T06:10:13.2572686Z �[96mLine |
2020-08-25T06:10:13.2573161Z �[96m 105 | �[0m �[96mWrite-AzOpsLog -Level Error -Topic "entrypoint" -Message �[0m …
2020-08-25T06:10:13.2573665Z �[96m | �[91m ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2020-08-25T06:10:13.2574164Z �[91m�[96m | �[91m[06:10:13.1439] (entrypoint) The property 'Properties' cannot
2020-08-25T06:10:13.2574644Z �[96m | �[91mbe found on this object. Verify that the property exists.
2020-08-25T06:10:13.2575024Z �[0m
2020-08-25T06:10:13.8864877Z Post job cleanup.

Steps to reproduce

1. Run AzOps-Pull

Screenshots

• Add Code Coverage Support

Describe the solution you'd like

Multi-language linter coverage for source code

Overview

To improve the quality of our code base, we should implement linting across all file types. A simple way to do this is to add github/super-linter to the build pipeline. This should provide coverage of most code types within our repository, with the primary exception of ARM Templates (which are validated by github/super-linter as standard JSON, using jsonlint).

To bridge the gap with ARM, we can incorporate Microsoft's own Azure Resource Manager Template Toolkit (arm-ttk).

Benefits

As documented by github/super-linter:

• Prevent broken code from being uploaded to the default branch
• Help establish coding best practices across multiple languages
• Build guidelines for code layout and format
• Automate the process to help streamline code reviews

And from azure/arm-ttk:

• Validating the author's intent (unused parameters or variables)
• Security practices for the language (outputting secrets in plain text)
• Using the appropriate language construct for the task at hand (using environmental functions instead of hard-coding values)

Prerequisites

To implement this feature, it will also be necessary to complete a clean-up of our source code so the linter tests do not fail, preventing the pipeline from successful completion.

Known Issues

Primary areas of concern for remediation include:

• Use of global variables within PowerShell functions;
• PSScriptAnalyzer throwing An item with the same key has already been added. Key: ResourceError errors when handling multi-threaded ForEach loops;

Other issues may also need further consideration before this feature can be fully implemented.

A manual run of Super-Linter has highlighted the following error statistics:

ERRORS FOUND in YML:[4]
ERRORS FOUND in JSON:[11]
ERRORS FOUND in MARKDOWN:[15]
ERRORS FOUND in DOCKER:[1]
ERRORS FOUND in POWERSHELL:[28]

It should be noted that these statistics represent the number of rules which have generated an error, and not the actual number of errors (which will be higher in some cases).

Describe the bug
Debug and Verbose are both set to "1" in azops.yml. AzOps Push Action now fails with the following error message returned:

Confirm
Are you sure you want to perform this action?
Performing the operation "Start-AzOpsNativeExecution" on target "Update
ErrorActionPreference to Continue?".
[Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help
Write-AzOpsLog: /action/entrypoint.ps1:53
Line |
53 | Write-AzOpsLog -Level Error -Topic "entrypoint" -Message …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| [2020-07-14 08:55:50.2249] (entrypoint) Exception calling
| "ShouldProcess" with "1" argument(s): "Object reference not
| set to an instance of an object."

(default is "Y"):

Steps to reproduce

1. Set Debug and Verbose in azops.yml to "1"
2. Create PR that invokes Push Action

Screenshots

Describe the bug
policySetDefinitions cannot be deployed, returned error is:

2020-08-20T09:18:22.3526536Z WARNING: [09:18:22.3506] (pwsh) effectiveResourceType: Microsoft.Authorization/policySetDefinitions AzOpsMainTemplate does NOT supports resource type Microsoft.Authorization/policySetDefinitions in /var/lib/app/template/template.json. Deployment will be ignored

Steps to reproduce

1. Create new policySetDefinitions on management group scope
2. Commit and create PR, which triggers AzOps-Push

Screenshots

Describe the bug

Attempted local docker build

Steps to reproduce

1. docker build --pull --rm -f "Dockerfile" -t azops:latest ".
   Sending build context to Docker daemon 3.173MB
   Step 1/18 : FROM mcr.microsoft.com/powershell:latest
   latest: Pulling from powershell
   Digest: sha256:9a141117590e8c7cad2abf42abee5c7a21e466679525c87d7a2455ad7d37e514
   Status: Image is up to date for mcr.microsoft.com/powershell:latest
   ---> f544cbdcb00a
   Step 2/18 : LABEL "com.github.actions.name"="AzOps"
   ---> Using cache
   ---> c1b86a0281e4
   Step 3/18 : LABEL "com.github.actions.description"="-"
   ---> Using cache
   ---> 49210f5e3518
   Step 4/18 : LABEL version="0.1"
   ---> Using cache
   ---> 7cb56272a59c
   Step 5/18 : LABEL repository="https://github.com/Azure/AzOps"
   ---> Using cache
   ---> f2755a3be839
   Step 6/18 : LABEL maintainer="Microsoft"
   ---> Using cache
   ---> c6330e4b7174
   Step 7/18 : ARG github=0.10.0
   ---> Using cache
   ---> 2f0dd80a6b3b
   Step 8/18 : ARG azure_accounts=1.8.1
   ---> Using cache
   ---> 69e9b44390f4
   Step 9/18 : ARG azure_resources=2.0.1
   ---> Using cache
   ---> efb62cb8f287
   Step 10/18 : RUN [ "/bin/bash", "-c", "apt-get update &> /dev/null && apt-get install -y git wget &> /dev/null" ]
   ---> Running in 78c7510aad8a
   The command '/bin/bash -c apt-get update &> /dev/null && apt-get install -y git wget &> /dev/null' returned a non-zero code: 100

Screenshots

Automated PR creation fails on forked repostory due to a missing system label.

1. Fork Azure/Enterprise-Scale repository
2. Ensure that you have cofigured AZURE_CERDENTIALS as a secrete
3. Start Discovery of your environment
4. Discovery will fail when creating the PR with the following message:

Creating pull request for system into main in Azure/Enterprise-Scale

could not add label: 'system' not found
Write-AzOpsLog: /action/entrypoint.ps1:93
Line |
  93 |              Write-AzOpsLog -Level Error -Topic "git" -Message $PSItem …
     |              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | [2020-06-22 14:29:02.6989] (git) Execution of ***               
     | gh pr create --title $env:INPUT_GITHUB_PULL_REQUEST --body
     | "Auto-generated PR triggered by Azure Resource Manager `nNew
     | or modified resources discovered in Azure" --label "system"   
     | *** by Invoke-AzOpsGitPull.ps1: line 99 failed with exit code 1

There is no mechanism to provide the Environment parameter to the various calls to Connect-AzAccount. This is necessary when the target environment is in one of the Microsoft private clouds. For example, without -Environment AzureUSGovernment AzOps cannot authenticate into the Azure government environment, as it attempts to authenticate against login.micosoftonline.com instead of login.microsoftonline.us. Forwarding to Azure private clouds from the commercial login endpoint was discontinued by a policy in 2017.

Describe the bug

Moving subscriptions via AzOps doesn't work.

Steps to reproduce

Run the 'deploy to azure' for Contoso which setup the policy definitions, mgmt group structure, la workspace and automation account and policy assignment for 'deploy-azactivity-log'.

Add existing subscription as child item of a mgmt group that was just created via template. Expectation is subscription is moved under mgmt group and activity log diag setting is created.

Actual result: Build is green but subscription is not moved by AzOps.

Spoke to @uday31in and he says there is a bug in the if statement here:

Screenshots

Describe the solution you'd like

Describe the bug

The Azure Pipelines status badge is incorrectly reporting on the README.md. Currently it's set to pull the status of the 'main' branch, in the repository we're building on pull requests though and so we need to fix this.

• Az.Resources
• Az.Accounts
• Pester

[Prototype]

• Develop an initial prototype for migrating to Docker commands instead of GitHub Actions
• Develop parity between GitHub / Azure DevOps tasks

Describe the bug

WARNING: Retry Count: 1 Caught Exception for Credential Error for Get-AzResource for /subscriptions/00000000-0000-0000-0000-00000000000/resourceGroups/user-keyvault

Steps to reproduce

1. Invoking the Initialize-AzOpsRepository cmdlet

WARNING: Breaking changes in the cmdlet 'Get-AzManagementGroup' :
WARNING: - The parameter : 'GroupName' is being replaced by parameter : 'GroupId'.

Replace instances of Get-AzManagementGroup -GroupName with -GroupId accross the codebase.

I deployed the Wingtip reference implementation accordingly

I want to move Landing Zone Subscription from MG ES-Online to ES-decommissioned by updating the root level management groups parameter json file

After PR the AzOps action in the AzOps-Push Github fails with the following error:

[11:28:00.8273] (git) Deployment required
[11:28:00.8321] (git) Add / Modify:
[11:28:00.8332] (git) azops/Tenant Root Group (496f0b27-4fa4-4c3d-8bbe-19c4b6875c81)/.AzState/Microsoft.Management_managementGroups-496f0b27-4fa4-4c3d-8bbe-19c4b6875c81.parameters.json
[11:28:00.8338] (git) Delete:
[11:28:00.9567] (pwsh) Template NOT found /github/workspace/azops/Tenant Root Group (496f0b27-4fa4-4c3d-8bbe-19c4b6875c81)/.AzState/Microsoft.Management_managementGroups-496f0b27-4fa4-4c3d-8bbe-19c4b6875c81.json
[11:28:00.9578] (pwsh) Determining resource type /var/lib/app/template/template.json
[11:28:00.9751] (pwsh) effectiveResourceType: /providers/Microsoft.Management/managementGroups AzOpsMainTemplate supports resource type /providers/Microsoft.Management/managementGroups in /var/lib/app/template/template.json
Write-AzOpsLog: /var/lib/app/entrypoint.ps1:105
Line |
 105 |              Write-AzOpsLog -Level Error -Topic "entrypoint" -Message  …
     |              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | [11:28:02.6682] (entrypoint) [11:28:02.6589]
     | (New-AzOpsDeployment) Template Validation Failed Unable to
     | edit or replace deployment
     | 'NorthStar-MG-496f0b27-4fa4-4c3d-8bbe-19c4b6875c81': previous
     | deployment from '9/11/2020 10:07:05 AM' is still active
     | (expiration time is '9/18/2020 10:07:04 AM'). Please see
     | https://aka.ms/arm-deploy for usage details.

When checking the Azure Portal deployments the following deployment were started

Why is a deployment called NorthStar-MG-496f0b27-4fa4-4c3d-8bbe-19c4b6875c81 started?

Describe the bug
The ARM Template link in the README files returns a 404

Steps to reproduce

1. Click on the link ;)

Screenshots

Describe the solution you'd like

Describe the solution you'd like

Update CI path to include template directory

It takes some time to build the container every time, could it be hosted on MCR?

This would also drive the release management process, which will become important as adoption grows.