Unable to reference script in same directory as reusable step in a repository about azure-pipelines-yaml HOT 27 CLOSED

This would be awkward to include in the language, given how the various features work. I think with the introduction of extends templates, that's the direction I'd push you. Something like:

parameters:
- name: usersteps
  type: stepList
  default: []

resources:
  repositories:
  - repository: tools
    type: git
    name: <place where this YAML is>

steps:
- checkout: tools
  path: s/tools
- checkout: self
  path: s/code
- ${{ parameters.usersteps }}

Given that we have multicheckout now, I'm marking this issue complete.

from azure-pipelines-yaml.

@afeblot thanks that helps. It's a good scenario. Unless @vtbassmatt seems something I don't, then I don't think we have immediate plans to handle it in a simple way.

I do find this scenario very interesting and we'll continue to think about how to accommodate the scenario.

We have kicked around an idea before, like:

# my-tools.yml

steps:

# Checkout the tools repository. In this proposal, 'currentRepository' resolves to the repository
# of the current file being parsed.
- checkout: ${{ currentRepository.alias }}

# iirc multi-checkout will sync the repo to a folder named using the alias by default.
# So basically i'm just using the alias to resolve the local path as it will be checked-out
# on the agent.
- bash: ${{ currentRepository.alias }}/setup-tools.sh

Let us know what you think.

from azure-pipelines-yaml.

Something else would be useful: The possibility for a template file to refer to another template file in a different git repo. Unless this change recently, a template file refuses to hold a resources key.
I'd use this to have a company-scope first level of templates which would be used by a number of team-scope second level of templates which would be used by these team pipelines.

# my-team-template.yml (in Contoso/teamXXXtemplates)

resources:
  repositories:
  - repository: companyCommonTemplates
    type: github
    name: Contoso/companyCommonTemplates

parameters: {}

stages:
- template: my-company-stages.yml@companyCommonTemplates
- stage: myTeamXXXStage1
- stage: myTeamXXXStage2

from azure-pipelines-yaml.

Having the possibility to reference a template from another repository which itself consume scripts in its own repository is a must.

As we move our release pipelines to YAML, we are relying more and more on PowerShell scripts instead of steps like AzureCLI. But inlinnng those scripts is not possible because of the length limitation but also because it prevents debugging scripts locally. By having the PowerShell in a separate file, we can debug it without having to pushing to the repo and running the entire pipeline.
I am not even talking about syntax coloring, IntelliSense with auto-completion, documentation and snippets...

We could use the multi-checkout as specified to retrieve both the YAML file(s) and their associated script(s) if they have some... But it requires we are tightly bound to how the YAML template is implemented. Ideally, we what the template to be independent so contributor can decide to inline or use a script or whatever resource without impacting template consumers.

Also scripts are not the only resource our templates depends on, we can also depends on configuration files, ARM template files, ...

In a nutshell, we want the YAML to be as dumb as possible as it is not as developer friendly :)

from azure-pipelines-yaml.

It would be so helpful that a template can reference files from its own repo with a syntax such as otherFile@self.

from azure-pipelines-yaml.

@vtbassmatt this issue is good feedback to think about. It sounds like it lines up with multi-checkout

from azure-pipelines-yaml.

Thanks, @ericsciple . Yes, we discovered on by our own a few days ago :-)
As we're on this, would there be some similar ways to reference an external (non-template) file from the template repository, like a powershell script that would be too long to be inlined in the template?

from azure-pipelines-yaml.

@CoolDadTx We currently do not have a model that lest you pull other files out of your template repo as the YAML files are processed on the service and not on the agent. For your scenario of reuse, I would recommend you inline the contents of your script in the YAML file.

from azure-pipelines-yaml.

@chrisrpatterson, inlining the script was the original plan but because of the other serious restrictions in YAML about template expression placement it became very difficult. I might have to wait for the updated version of YAML that allows template expressions anywhere in the template.

from azure-pipelines-yaml.

Note, support for template expressions embedded within strings is rolling out with the next sprint's payload. The deployment starts next week, and takes about 3 weeks to reach all accounts.

from azure-pipelines-yaml.

You could also map the template expressions into the env section on the script and then reference the environment variables.

from azure-pipelines-yaml.

@ericsciple hey, any updates? how does this work (does it?) examples? also, can I do something like this:

    Inline: |
        ${{ format('bla-bla-bla
        bla-bla-bla', parameters.param1, parameters.param2) }}

doesnt seem to work for me. keeping 500 characters on 1 line is a pain

from azure-pipelines-yaml.

this works for me:

pool: hosted ubuntu 1604
steps:
- bash: |
    echo ${{ 'hello single line' }}
    echo ${{ '''hello
    multiline''' }}

from azure-pipelines-yaml.

ok, but what about format function? i thought i'm supposed to start it with ${{ do i just use one per line? @ericsciple
ok, my initial example works, not sure why it didnt work first time

from azure-pipelines-yaml.

Here is how the format function works. It looks like it never made it into the official docs... I'll fix that.

You can now use as many ${{ }} expressions inline as you want. So the need for the format function was basically eliminated for this situation. You can still use it if you want. For example, this works:

pool: hosted ubuntu 1604
steps:
- script: |
    echo "Hello ${{ format('{0} ({1})', variables['build.requestedFor'], variables['build.requestedForEmail']) }} and welcome to ${{ variables['system.teamProject'] }}!"

from azure-pipelines-yaml.

@afeblot within the same repo, don't use a suffix and it works.

For example:

# entry file

resources:
  repositories:
  - repository: other
     [...]
steps:
- template: my-steps.yml@other

# my-steps.yml

steps:
- template: my-pre-steps.yml

# my-pre-steps.yml

steps:
- script: echo hello from my-pre-steps.yml

from azure-pipelines-yaml.

@afeblot checkout the multi-checkout spec I linked above (I fixed the link now).

That would make it easy to sync an external repo and run a script within in it.

What it doesn't solve is, wrapping up that behavior into a template that is also defined in the external repo. Is that the behavior you are after? Or would making it easy to sync the the external repo, suffice?

from azure-pipelines-yaml.

@ericsciple This multi-checkout seems nice but I suppose it can't help for my use case where the template is in its own git repo, and this template would need to execute additional scripts or read additional files from this template repo.

My current ugly workaround: I just do it with the basic tools: a template task does git clone the template repo to access such files, but that requires me to hardcode the repo url, and the branch to checkout:

# in my template definition:

    - bash: |
        cd $(Build.SourcesDirectory)
        git clone https://${SYSTEM_ACCESSTOKEN}@xxxxxxx.visualstudio.com/MYPROJECT/_git/template_repo
      env:
        SYSTEM_ACCESSTOKEN: $(System.AccessToken)
      displayName: 'Clone template repository'

    - bash: |
        do something with template_repo/some_file

Would there be any way to benefit from these repository definitions and checkout tasks directly in a template to make this git clone cleaner/simpler?

from azure-pipelines-yaml.

@ericsciple that would be perfect.

Though, maybe self instead of currentRepository, to be consistent with the multi-checkout proposal?
Unless self is supposed to refer to the repo containing the entry file even when used in the template file?

from azure-pipelines-yaml.

Any update with this? Also looking to ref a file in a template repo

from azure-pipelines-yaml.

Thanks for your interest. No updates at the moment. It's on my radar but probably won't bubble to the top for at least a few months. We need to get multi-checkout done (which itself is blocked behind other stuff... but you don't care about the minutiae of my backlog 😁) first.

from azure-pipelines-yaml.

ok thanks for the information! look forward to multi-checkout!

from azure-pipelines-yaml.

Hello,

any updates of these functionalities? :) when we are pointing on a resource (repo), can we only refer to an yaml file?

Thanks,
Rafal

from azure-pipelines-yaml.

We're adding multi-repo checkout this sprint. Then you'll be able to do something like:

resources:
  repositories:
  - repository: other
    name: MyProject/MyOtherRepo

steps:
- checkout: other
- checkout: self
- template: somesteps.yml@other

Notice that we're both referring to a template and checking out the repo called other.

Edit: and to be clear, this will ship in mid-November 2019.

from azure-pipelines-yaml.

Thanks @vtbassmatt for replying! Great update then. It means that we will be able to reuse not only yaml files, but also other files that are in, lets say, 'other' repository.

from azure-pipelines-yaml.

Nice 👍
@vtbassmatt, do you still have on your radar the wider use case we discussed here on Feb 6 (doing the same directly from a template file) ?

from azure-pipelines-yaml.

Yes @afeblot, that'll work too.

from azure-pipelines-yaml.