sam-cogan / arm-ttk-extension-xplatform Goto Github PK

It was noted here.
Azure/azure-cli#25710

The fix was mentioned as potentially coming in with this PR scheduled for deployment today, but if your useAzBicep doesn't pull the latest version then it might still fail with the error mentioned above (and an example below).
Azure/azure-cli#25729 (comment)

Using executable 'pwsh'
pwsh ps-runner.ps1 -templateLocation AzureResources\* -resultLocation TestResults -skipTests <snipped> `
 -mainTemplates main.bicep -cliOutputResults -ignoreExitCode -recurse -useAzBicep
##[error]ERROR: The command failed with an unexpected error. Here is the traceback:
ERROR: No section: 'bicep'
Traceback (most recent call last):
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 233, in invoke
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 663, in execute
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 697, in _run_job
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 333, in __call__
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 121, in handler
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/resource/custom.py", line 3671, in build_bicep_file
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/resource/_bicep.py", line 85, in run_bicep_command
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/resource/_bicep.py", line 141, in ensure_bicep_installation
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/config.py", line 99, in get
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/config.py", line 94, in get
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/config.py", line 208, in get
  File "configparser.py", line 783, in get
  File "configparser.py", line 1154, in _unify_values
configparser.NoSectionError: No section: 'bicep'

Just migrated to the xplat version of this awesome extension. Noticed that it uses 0.17 of ARM-TTK and that the latest version is 0.22, https://github.com/Azure/arm-ttk/releases. Seeing from the release on ARM-TTK there have been 5 versions in around 3 months.

To get better support for newer versions of ARM-TTK, I suggest that the extension provides a flag to use latest version of ARM-TTK instead of the embedded version.

I can see that its done for the bicep executable, so why not download latest from https://aka.ms/arm-ttk-latest ?

Hi,

Been using this extension for a couple of months, and it has been really useful so far. I've started to look into using a bicep registry in a private container registry but get failures as the task has no concept of being logged into Azure. Would it be possible to include a parameter for specifying an Azure Subscription to perform an az login and provide access to scanning referenced bicep modules?

Many Thanks
Cameron

Starting to use the new cross platform version.

In the 'old' version bicep linter suppres rules seemed to be evaluated, i.e.

#disable-next-line secure-secrets-in-params
param SomeParamWithSecretInName string 

But now this does not seem to be evaluated any longer? - getting error
Parameter "SomeParamWithSecretInName" is of type "string" but should be secure. in template file main.json

even though there is a suppress statement above.

I was just wondering if this extension supports bicep files directly? Or do we have to compile to an arm template before hand?

If you run from PowerShell and from the pipeline, PowerShell will return a "recommended" or "warning" result while the extension will mark it as passed. Can we differentiate those in the test result files?

I have my template in $(System.DefaultWorkingDirectory)/Terraform Storage, which will not work:

pwsh /home/vsts/work/_tasks/RunARMTTKTestsXPlat_cb27e003-8534-4da9-aaa0-35ca8756ae29/1.2.0\powershell\ps-runner.ps1 -templateLocation /home/vsts/work/1/s/Terraform Storage -resultLocation /home/vsts/work/1/s/results -cliOutputResults -ignoreExitCode -recurse
##[error]The input file path cannot be specified multiple times

##[error]The input file path cannot be specified multiple times

Issue

I have just migrated from the Windows PowerShell to this Cross Platform version. The only change I thought we required was in the task name swapping to

- task: RunARMTTKTestsXPlat@1
  displayName: "Run ARM TTK tests on built ARM"
  inputs:
    templatelocation: '$(Build.SourcesDirectory)\BicepTemplates\ARMoutputs\*.json'
    resultLocation: '$(System.DefaultWorkingDirectory)\results'
    skipTests: '$(TestsToSkip)'
    cliOutputResults: true

However I have found that if have a comma separated list for -skipTests we get the following error

Using executable 'powershell.exe'
powershell.exe E:\Agent_work_tasks\RunARMTTKTestsXPlat_cb27e003-8534-4da9-aaa0-35ca8756ae29\1.2.1\powershell\ps-runner.ps1 -templateLocation E:\Agent_work\17\s\BicepTemplates\ARMoutputs*.json -resultLocation E:\Agent_work\17\s\results -skipTests VM Images Should Use Latest Version,Resources Should Have Location,Location Should Not Be Hardcoded,Template Should Not Contain Blanks,DeploymentTemplate Must Not Contain Hardcoded Uri,Outputs Must Not Contain Secrets,URIs Should Be Properly Constructed -cliOutputResults -recurse
##[error]E:\Agent_work_tasks\RunARMTTKTestsXPlat_cb27e003-8534-4da9-aaa0-35ca8756ae29\1.2.1\powershell\ps-runner.ps1 : Cannot

##[error]E:\Agent_work_tasks\RunARMTTKTestsXPlat_cb27e003-8534-4da9-aaa0-35ca8756ae29\1.2.1\powershell\ps-runner.ps1 : Cannot

##[error]process argument transformation on parameter 'clientSecret'. Cannot convert value to type System.String.
At line:1 char:265
... ults -skipTests VM Images Should Use Latest Version,Resources Should ...
CategoryInfo : InvalidData: (:) [ps-runner.ps1], ParameterBindingArgumentTransformationException
FullyQualifiedErrorId : ParameterArgumentTransformationError,ps-runner.ps1

Analysis

Experimentation of running the ps-runner.ps1 from the command prompt on a self hosted build agent showed that the solution was to enclose the skip test list for the inner PowerShell running in quotes.

powershell\ps-runner.ps1 -templateLocation E:\Agent\_work\17\s\BicepTemplates\ARMoutputs\*.json -resultLocation E:\Agent\_work\17\s\results -skipTests 'VM Images Should Use Latest Version,Resources Should Have Location,Location Should Not Be Hardcoded,Template Should Not Contain Blanks,DeploymentTemplate Must Not Contain Hardcoded Uri,Outputs Must Not Contain Secrets,URIs Should Be Properly Constructed' -cliOutputResults -recurse

Once this was done ARM TTK ran as expected.

Have I missed something related to parameter passing, am I passing in the list incorrectly?

I will link a PR with this change to this issue on the assumption this fix is valid. I have assumed the same fix is needed for the includeTests parameter

When using the skipTests input the list of tests doesn't seem to be passing through correctly.

- task: RunARMTTKTestsXPlat@1
  inputs:
   templateLocation: infra.json
   skipTest: apiVersions Should Be Recent
   allTemplatesMain: false

Running this I get an error in the pipeline and the test is not skipped

WARNING: Test 'Should' was not found, all tests will be run

I've tried a couple different tests with the same issue. Seems to error with the second word of the test. I've also tried with and without quotes and with the test name all lower case. Any ideas?

If there is an autogenerated ARM template that you want to skip test validation on, it would be nice if there was a way to mute some or all tests on just that file.

Our current work around is to put it in a different directory from all our other ARM templates, but it would be preferable to either be able to mute tests within a file, or exclude one file.

Something like // arm-ttk-disable-file ... at the top of the file would be nice

I have an issue in a rather advanced bicep template, consisting of a main template with a dozen child templates. I am using the latest version of the ARM-TTK xplat extension (v1.2.0) and using the az bicep option (with bicep extension being up-to-date).

The error I am seeing is:

##[error]Write-Error: C:\agents\agent01\_work\_tasks\RunARMTTKTestsXPlat_cb27e003-8534-4da9-aaa0-35ca8756ae29\1.2.0\arm-ttk\Test-AzTemplate.ps1:785
##[error]Write-Error: C:\agents\agent01\_work\_tasks\RunARMTTKTestsXPlat_cb27e003-8534-4da9-aaa0-35ca8756ae29\1.2.0\arm-ttk\Test-AzTemplate.ps1:785
##[error]Line |
##[error]Line |
##[error] 785 |                  Test-FileList # we just call it directly.
##[error] 785 |                  Test-FileList # we just call it directly.
##[error]     |                  ~~~~~~~~~~~~~
##[error]     |                  ~~~~~~~~~~~~~
##[error]     | Could not extract inner templates for 'C:\agents\agent01\_work\6\s\templates\bicep\maintemplate.json'.
##[error]     | Could not extract inner templates for 'C:\agents\agent01\_work\6\s\templates\bicep\maintemplate.json'.
##[error]
##[error]

I have set the following flags for the task:

     - task: RunARMTTKTestsXPlat@1
        displayName: Run ARM TTK Tests
        inputs:
          templatelocation: '$(Build.SourcesDirectory)/templates/bicep'
          resultLocation: '$(Build.SourcesDirectory)/ttk-results'
          skipTests: 'Parameter Types Should Be Consistent,URIs Should Be Properly Constructed'
          mainTemplates: 'maintemplate.bicep'
          allTemplatesMain: false
          cliOutputResults: false
          ignoreExitCode: true
          usePSCore: true
          useAzBicep: true

But the task still fails even though I set ignoreExitCode: false and the task also outputs the same no matter what I set cliOutputResults to.

Any thoughts on what could be done to not have it fail. The bicep templates are compiled just fine, the reason for supressing tests is that bicep generated ARM template contains some constructs that ARM TTK does not allow, so we just ignore those.

As title implies, this extension needs a bump to arm-ttk 0.23

It would be great if we could point it to a folder with additional custom tests.

Issue

When this task is run within an Azure DevOps pipeline a Windows agent the following error is sometimes seen

##[error]Import-Json : Import failed for
'E:\Agent_work\92\s\Src\Expenses\BlackMarble.ISS.Bicep\ARMOutput\RootTemplate-Main.json': Exception calling "Replace"
with "2" argument(s): "The RegEx engine has timed out while trying to match a pattern to an input string. This can
occur for many reasons, including very large inputs or excessive backtracking caused by nested quantifiers,
back-references and other factors." Exception calling "Replace" with "2" argument(s): "The RegEx engine has timed out
while trying to match a pattern to an input string. This can occur for many reasons, including very large inputs or
excessive backtracking caused by nested quantifiers, back-references and other factors." Exception calling "Replace"
with "2" argument(s): "The RegEx engine has timed out while trying to match a pattern to an input string. This can
occur for many reasons, including very large inputs or excessive backtracking caused by nested quantifiers,

I suspect the issue is down to the complexity of the ARM template being analyzed

Analysis

If I RDP onto my on prem self hosted agent VM I can get the same error if I follow the following process

From Command Prompt with PowerShell

1. Open a Command Prompt (CMD)
2. Run the command powershell.exe E:\Agent\_work\_tasks\RunARMTTKTestsXPlat_cb27e003-8534-4da9-aaa0-35ca8756ae29\1.2.2\powershell\ps-runner.ps1 -templateLocation E:\Agent\_work\92\s\Src\Expenses\BlackMarble.ISS.Bicep\ARMOutput\* -resultLocation E:\Agent\_work\92\s -recurse
3. The same error as seen in a pipeline run is seen

From Command Prompt with PowerShell Core

1. Open a Command Prompt (CMD)
2. Run the command pwsh.exe E:\Agent\_work\_tasks\RunARMTTKTestsXPlat_cb27e003-8534-4da9-aaa0-35ca8756ae29\1.2.2\powershell\ps-runner.ps1 -templateLocation E:\Agent\_work\92\s\Src\Expenses\BlackMarble.ISS.Bicep\ARMOutput\* -resultLocation E:\Agent\_work\92\s -recurse
3. The script runs as expect (I do have ARM TTK issue with the file under test)

From PowerShell 5 Prompt

1. Open a PowerShell 5 Prompt
2. Run the command E:\Agent\_work\_tasks\RunARMTTKTestsXPlat_cb27e003-8534-4da9-aaa0-35ca8756ae29\1.2.2\powershell\ps-runner.ps1 -templateLocation E:\Agent\_work\92\s\Src\Expenses\BlackMarble.ISS.Bicep\ARMOutput\* -resultLocation E:\Agent\_work\92\s -recurse
3. The script fails as expected

From PowerShell 7 Prompt

1. Open a PowerShell 7 Prompt
2. Run the command E:\Agent\_work\_tasks\RunARMTTKTestsXPlat_cb27e003-8534-4da9-aaa0-35ca8756ae29\1.2.2\powershell\ps-runner.ps1 -templateLocation E:\Agent\_work\92\s\Src\Expenses\BlackMarble.ISS.Bicep\ARMOutput\* -resultLocation E:\Agent\_work\92\s -recurse
3. The script runs as expected (I do have ARM TTK issue with the file under test)

Suggestion

The issue appears to be the use of the PowerShell 5 (powerhell.exe) as opposed to PowerShell 7 (pwsh.exe). I think the best option is to give the user the choice to force the use of PWSH, whilst leaving the current default behavior as it is

I will create a PR with this change

Note: This maybe replaced to the problem (arm-ttk-extension #10)[https://github.com/sam-cogan/arm-ttk-extension/issues/10)

Currently bicep.exe is downloaded (if not present on agent) when analyzing bicep templates.

I would like the option to instead use az bicep which could be available on the agent through the azure cli. This would allow me to control bicep updates using az bicep upgrade in my pipelines and be sure that whatever tasks runs before and after ARM-TTK xplat tasks use same version of bicep as the ARM-TTK task.

It could be a simple flag under advanced, similiar to usePsCore with a default value of false, but if set to true, then calls to bicep.exe is replaced with az bicep.

If would be up to pipeline developer to ensure Azure CLI is installed prior to running the ARM-TTK task.

Currently, when some of test are failing then only useful information that is printed out is "#[error]Failures found in test results" even debug is enabled on pipeline. You need to go to Tests tab to find errors description.
This behavior is OK, for default option, but it would be nice with some input task parameter(e.g. 'detailInfo') to control how much detailed is displayed in pipeline output.
This is especially nice when new people are added to project, and it is not immediately known that you need to go to Tests tab to find out what are errors about. If you only click on failing task in pipeline you will not see to much info :-).

The link currently points to https://github.com/Azure/azure-quickstart-templates/tree/master/test/arm-ttk

Please adjust.