Paramaterise task files about concourse-tutorial HOT 19 CLOSED

Here is an example job file that uses params:

---
platform: linux
image_resource:
  type: docker-image
  source: {repository: dingotiles/dingo-postgresql-release-pipeline}

inputs:
- name: boshrelease-ci
- name: etcd
- name: simple-remote-syslog
- name: candidate-release

outputs:
- name: manifest

params:
  bosh_target:
  bosh_username: admin
  bosh_password: admin
  bosh_syslog_host:
  bosh_syslog_port:
  deployment_name:
  docker_image_source: embedded
  docker_image_image: "dingotiles/dingo-postgresql95"
  docker_image_tag: "9.5"
  aws_access_key:
  aws_secret_key:
  s3_endpoint:
  region:
  backups_bucket:
  clusterdata_bucket:

run:
  path: boshrelease-ci/ci/tasks/create-manifest.sh

When a vaule is specified like docker_image_source: embedded this will be the default value. Setting the param explicitly from the pipeline.yml will override the default.

Here is the correspoding section of the pipeline.yml:

  - task: create-lite-manifest
    file: boshrelease-ci/ci/tasks/create-manifest.yml
    params:
      deployment_name: dingo-postgresql-testflight
      bosh_target: {{bosh-lite-target}}
      bosh_username: {{bosh-lite-username}}
      bosh_password: {{bosh-lite-password}}
      bosh_syslog_host: {{bosh-lite-syslog-host}}
      bosh_syslog_port: {{bosh-lite-syslog-port}}
      aws_access_key: {{dingotiles-aws-access}}
      aws_secret_key: {{dingotiles-aws-secret}}
      backups_bucket: {{backups-testflight-backups-bucket}}
      clusterdata_bucket: {{backups-testflight-clusterdata-bucket}}
      s3_endpoint: {{backups-testflight-s3-endpoint}}
      region: {{backups-testflight-s3-region}}
      docker_image_image: "dingotiles/dingo-postgresql95"
      docker_image_tag: "edge"
      docker_image_source: upstream

Hope that helps

from concourse-tutorial.

Thanks for your help.

We want to use a private Docker Hub repo in task yml config without hardcoding credentials. For pipeline ymls I use {{parameters}} and then inject them with --load-vars-from.

I thought fly merge this parameters in the pipeline ymls on the set-pipeline, but our task files are in our repo. Because of that they are not available when the task started..

from concourse-tutorial.

To confirm I think the one part you want to configure is the location of the task image. You're right - this isn't parametrizable afaik. You'll code this into your task.yml.
If it is possible, or if it's a feature you need, ask the Concourse core team on their slack account

On Wed, Jun 1, 2016 at 7:35 AM -0700, "Yeray Medina López" [email protected] wrote:

Thanks for your help.

We want to use a private Docker Hub repo in task yml config without hardcoding credentials. For pipeline ymls I use {{parameters}} and then inject them with --load-vars-from.

I thought fly merge this parameters in the pipeline ymls on the set-pipeline, but our task files are in our repo. Because of that they are not available when the task started..

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

from concourse-tutorial.

Ahh, ok so I don't think task: test params get passed into /image_resource/source, rather they only get passed into params and ultimately are forwarded into the env vars of the /run/path command.

Can I ask you escalate this request to the concourse team for this?

In the meantime, I propose you flatten your test.yaml into the pipeline.yaml. I typically do this for all my personal pipelines (that is, I don't have test.yml files, rather everything is in pipeline.yml).

jobs:
- name: development
  serial: true
  plan:
  - get: git-repo
    trigger: true

  - task: test
    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          repository: super-secret.azurecr.io/drone-toolchain
          username: ((REG_USERNAME))
          password: ((REG_PASSWORD))

      inputs:
      - name: git-repo

      run:
        path: /bin/bash
        args:
        - -c
        - |
          cd git-repo
          echo "Hello World from my tests"

from concourse-tutorial.

Hello,

I have the same issue that is described above and asked about it in slack:

I have an issue with a concourse pipeline definition. I have a pipeline YAML with configuration variables (that are read corretcly) and this file references a task in another YAML file. From this file, I cannot read the configuration variables in the pipeline's scope. I have created an example of this case here: https://gist.github.com/ymedlop/fe7d9804684971cb24d3f091119acbed Is there a way for accessing the variables from the second YAML file?

from concourse-tutorial.

Thanks for your reply!.

We already opened a issue here

from concourse-tutorial.

Sounds like @ymedlop is all set. @darrenhaken did you get enough info for your question?

from concourse-tutorial.

@darrenhaken ^^

from concourse-tutorial.

All sorted @xanderrman

from concourse-tutorial.

thank you so much @darrenhaken 👍

from concourse-tutorial.

@bodymindarts And where are placed variables you sent in the target docker image?
In env variables? In a file?
Thanks !

from concourse-tutorial.

The answer to my question is here.

from concourse-tutorial.

Doesn't work for me

      - task: task-deploy-vm
        file: oe-tune-git-repo/ci/03-task-deploy-vm.yml
        params:
          SERVER: {{testing_deployment_server}}
          SSH_PUB_KEY: {{testing_deployment_server_ssh_pub_key}}
          SSH_PRIV_KEY: {{rsync_private_repo_key}}
          SSH_PASSWORD: {{testing_deployment_server_ssh_password}}

My script in my container

if [ -z ${SERVER} ]; then echo "SERVER is unset" && exit 1; else echo "SERVER is set to '$SERVER'"; fi
if [ -z ${SSH_PUB_KEY} ]; then echo "SSH_PUB_KEY is unset" && exit 1; else echo "SSH_PUB_KEY is set to '$SSH_PUB_KEY'"; fi
if [ -z ${SSH_PRIV_KEY} ]; then echo "SSH_PRIV_KEY is unset" && exit 1; else echo "SSH_PRIV_KEY is set to '$SSH_PRIV_KEY'"; fi
if [ -z ${SSH_PASSWORD} ]; then echo "SSH_PASSWORD is unset" && exit 1; else echo "SSH_PASSWORD is set to '$SSH_PASSWORD'"; fi

Output:

+ '[' -z ']'
+ echo 'SERVER is unset'
SERVER is unset
+ exit 1

It work only if I place variable in the task file. And I cannot use brackets to load the variables from a loaded yaml file.
This is ok in the task file:

params:
  SERVER: test
  SSH_PUB_KEY: test
  SSH_PRIV_KEY: test
  SSH_PASSWORD: test

Yen I set this

params:
          SERVER: {{testing_deployment_server}}
          SSH_PUB_KEY: {{testing_deployment_server_ssh_pub_key}}
          SSH_PRIV_KEY: {{rsync_private_repo_key}}
          SSH_PASSWORD: {{testing_deployment_server_ssh_password}}

I get this output

error: yaml: invalid map key: map[interface {}]interface {}{"testing_deployment_server":interface {}(nil)}

from concourse-tutorial.

Same error here. Do we have any idea how we can fix this? Any workaround?

from concourse-tutorial.

@koalalorenzo @Sispheor in your task file (e.g. 03-task-deploy-vm.yml) did you specify each of the params:` that you want to pass thru to the container script?

That is, in 03-task-deploy-vm.yml you would need a root-level params: with each key mentioned, with an optional default value:

params:
  SERVER:
  SSH_PUB_KEY:
  SSH_PRIV_KEY:
  SSH_PASSWORD: some-default

from concourse-tutorial.

Yes I did, I have it in the params of the tasks as well as passed on the pipe line list of tasks.

Please do not close the issue.

from concourse-tutorial.

@koalalorenzo can I ask a favour? can you make a mini gist with a pipeline, task.yml, and task.sh that reproduces the problem?

from concourse-tutorial.

Here: https://gist.github.com/koalalorenzo/749b40b70e1d0b40b11fa89bb9361fdd

from concourse-tutorial.

Done :) Please feel free to add more comments there if I missed some info. (it is a short issue with a link to this one :O )

from concourse-tutorial.