Expose YAML validation features as standalone tool about azure-pipelines-yaml HOT 33 CLOSED

Yes, it's on our short backlog to create a cli

from azure-pipelines-yaml.

@vtbassmatt This works great, I made a powershell cmdlet for it.
https://gist.github.com/JustinGrote/4ece827540da00d5ffce1d9eb593e56c

One thing: The most useful thing for this would be testing templates to make sure they expand properly. Is there any way in the YamlOverride to specify multiple files or somehow test templates?

from azure-pipelines-yaml.

We have shipped a server-side API for this. No plans to ship a fully standalone version, though, because the amount of server logic we would have to reimplement is too great.

POST to dev.azure.com/<org>/<project>/_apis/pipelines/<pipelineId>/runs?api-version=5.1-preview with a JSON body like this:

{
  "PreviewRun": true,
  "YamlOverride": "
# your new YAML here, optionally
"
}

The response will contain the rendered YAML.

from azure-pipelines-yaml.

@4c74356b41 Build - read & execute

from azure-pipelines-yaml.

@JustinGrote Any plans on publishing this to PowerShell modules gallery?

+1 on multi-file support as we heavily use YAML templates

from azure-pipelines-yaml.

@vtbassmatt thank you, and FYI while this is a welcome change, the inability to resolve/test templates (which requires a separate file unless theres some magic you can bestow upon me) kinda limits its usefulness, since expanding ${{ each and ${{ if statements is my primary big use case for this vs. commit over-and-over.

from azure-pipelines-yaml.

FYI, related pull request to add this to the VSTeam Powershell Module
MethodsAndPractices/vsteam#272

from azure-pipelines-yaml.

+1 on this - Preferably via a CLI which we can pull in via NPM and maybe even a Azure DevOps task as well.

from azure-pipelines-yaml.

Did this ever get built?

from azure-pipelines-yaml.

That's a good tool indeed and would certainly help!

However, a standalone tool is a must as it should be able to run as part of a CI build.

from azure-pipelines-yaml.

No plans to implement a task. By default, the build identity doesn't have permissions to queue a dry-run build. But if you alter that, it's a relatively short shell script:

pool:
  vmImage: ubuntu-latest

steps:
- checkout: none
- bash: |
    RUNS_URL=${SYSTEM_TEAMFOUNDATIONCOLLECTIONURI}${SYSTEM_TEAMPROJECT}/_apis/pipelines/${SYSTEM_DEFINITIONID}/runs?api-version=5.1-preview
    echo $RUNS_URL
    curl --user PAT:$(System.AccessToken) \
      -sS \
      --header "Content-Type: application/json" \
      --request POST \
      --data '{"PreviewRun": true,"YamlOverride":"steps:[{checkout:none}]"}' \
      $RUNS_URL

(Obviously replacing the payload with your chosen YAML.)

from azure-pipelines-yaml.

@JustinGrote that cmdlet is great, thanks for sharing! We don't have a multi-file version of this endpoint right now, but feature request noted.

from azure-pipelines-yaml.

We are also using templates heavily, but have centralised them in a single repository. I've now setup a build pipeline for that repository to validate all the YAML pipelines that use that repository with the new changes being made to it. This allows us to validate the changes before merging them which could potentially break all our pipelines. Not anymore with this feature which is nice. Maybe I should write a blog post ;).

from azure-pipelines-yaml.

@tomkerkhove it looks like @SebastianSchuetze is going to add it to the https://github.com/DarqueWarrior/vsteam module based on a twitter exchange we had :)

from azure-pipelines-yaml.

So I've been testing this out and this seems to only work if your entire pipeline definition is in a single file, and/or you are only trying to validate the pipeline entry point file, which doesn't work for us. We generally use templates, and as soon as you do that, it doesn't seem like it's possible for those to be picked up. If I make a typo in my local template file and then run this script, it succeeds and I see that my local files aren't being used in the expanded yaml. Is this scenario not supported?

from azure-pipelines-yaml.

Another thing which can be done is to make it like Gitlab - you can create an online validation tool (which is basically just a textarea) in the Azure Pipelines web app. The user will paste their yaml, and it'll show all the validation errors.

from azure-pipelines-yaml.

/cc @gopinathch

from azure-pipelines-yaml.

@kmahone i know you’d appreciate this!!

from azure-pipelines-yaml.

Is there any progress on providing this tool. The tool should help you validate and test your yaml instead of running your pipelines all the time.

from azure-pipelines-yaml.

Since it has not been mentioned, the VSCode extension does at least some local validation (I get red squiggles upon using invalid keywords or indentation): https://marketplace.visualstudio.com/items?itemName=ms-azure-devops.azure-pipelines

(the extension appears to use ajv, which I tried to run manually, via pajv, but got some errors -- for a pipeline schema which otherwise builds successfully. Probably just a configuration issue. Have not dug further as the extension is working great for me - thanks!)

from azure-pipelines-yaml.

Awesome, thanks!

@vtbassmatt Are there plans to provide a task for this at least?

It's a bit cumbersome to re-implement this for every customer.

from azure-pipelines-yaml.

Thanks!

from azure-pipelines-yaml.

@vtbassmatt noted. What I would recommend is that since yaml supports multiple document syntax with '---', to be able to add a comment or something to indicate which file it would be (by file path), that way the document and all its related sub templates and whatnot could be supplied as one entry to YamlOverride

from azure-pipelines-yaml.

If YAML templates are supported then I'm happy - Will give it a go.

Feel free to write that post, happy to share.

from azure-pipelines-yaml.

Great to hear, thanks!

from azure-pipelines-yaml.

@tomkerkhove I am currently sitting at the code for the PR

@JustinGrote do you have a direct link to the API documentation. It's kinda hard to find for me.

from azure-pipelines-yaml.

@SebastianSchuetze it's part of the preview API docs but I can't find the link anymore, however both of these reference the info:

#34 (comment)
https://docs.microsoft.com/en-us/azure/devops/release-notes/2020/pipelines/sprint-165-update#preview-fully-parsed-yaml-document-without-running-the-pipeline

Also if you look at my source code it's laid out there too.

from azure-pipelines-yaml.

We're working on getting the API docs to accurately reflect this (and other Pipelines endpoints).

from azure-pipelines-yaml.

So I've been testing this out and this seems to only work if your entire pipeline definition is in a single file, and/or you are only trying to validate the pipeline entry point file, which doesn't work for us. We generally use templates, and as soon as you do that, it doesn't seem like it's possible for those to be picked up. If I make a typo in my local template file and then run this script, it succeeds and I see that my local files aren't being used in the expanded yaml. Is this scenario not supported?

@vtbassmatt already commented on this. the short answer is "not yet"

#34 (comment)

from azure-pipelines-yaml.

@JustinGrote - thanks! I missed that (somehow)

from azure-pipelines-yaml.

Has something changed very recently? I've been trying this API for the first time today, and neither curl nor the PowerShell function provided by @JustinGrote work for me.

• Authorization for this API (not other DevOps APIs) only works if the username part of the HTTP Basic creds is left blank.
• The response is a redirect to a web page. If run through a browser, I end up at https://aex.dev.azure.com/me?mkt=en-US

from azure-pipelines-yaml.

@joergjo that sounds like an authentication or authorization problem; nothing in our API would trigger that behavior. You can call other APIs without a problem, though?

from azure-pipelines-yaml.

what scopes do I need on the PAT for this to work? the bare minimum, obviously. thanks

from azure-pipelines-yaml.