Giter VIP home page Giter VIP logo

concourse-pipeline-resource's Introduction

Concourse Pipeline Resource (DEPRECATED)

This resource is deprecated in favor of the set_pipeline step. No new versions will be released.

If there are limitations to the set_pipeline step that don't let you switch to it right now, please let us know in concourse/rfcs#31

Get and set concourse pipelines from concourse.

Installing

Use this resource by adding the following to the resource_types section of a pipeline config:

---
resource_types:
- name: concourse-pipeline
  type: docker-image
  source:
    repository: concourse/concourse-pipeline-resource

See concourse docs for more details on adding resource_types to a pipeline config.

Source configuration

Check returns the versions of all pipelines. Configure as follows:

---
resources:
- name: my-pipelines
  type: concourse-pipeline
  source:
    target: https://my-concourse.com
    insecure: "false"
    teams:
    - name: team-1
      username: some-user
      password: some-password
    - name: team-2
      username: other-user
      password: other-password
  • target: Optional. URL of your concourse instance e.g. https://my-concourse.com. If not specified, the resource defaults to the ATC_EXTERNAL_URL environment variable, meaning it will always target the same concourse that created the container.

  • insecure: Optional. Connect to Concourse insecurely - i.e. skip SSL validation. Must be a boolean-parseable string. Defaults to "false" if not provided.

  • teams: Required. At least one team must be provided, with the following parameters:

    • name: Required. Name of team. Equivalent of -n team-name in fly login command.

    • username: Basic auth username for logging in to the team. If this and password are blank, team must have no authentication configured.

    • password: Basic auth password for logging in to the team. If this and username are blank, team must have no authentication configured.

in: Get the configuration of the pipelines

Get the config for each pipeline; write it to the local working directory (e.g. /tmp/build/get) with the filename derived from the pipeline name and team name.

For example, if there are two pipelines foo and bar belonging to team-1 and team-2 respectively, the config for the first will be written to team-1-foo.yml and the second to team-2-bar.yml.

---
resources:
- name: my-pipelines
  type: concourse-pipeline
  source: ...

jobs:
- name: download-my-pipelines
  plan:
  - get: my-pipelines

out: Set the configuration of the pipelines

Set the configuration for each pipeline provided in the params section.

Configuration can be either static or dynamic. Static configuration has the configuration fixed in the pipeline config file, whereas dynamic configuration reads the pipeline configuration from the provided file.

One of either static or dynamic configuration must be provided; using both is not allowed.

static

---
resources:
- name: my-pipelines
  type: concourse-pipeline
  source:
    teams:
    - name: team-1

jobs:
- name: set-my-pipelines
  plan:
  - put: my-pipelines
    params:
      pipelines:
      - name: my-pipeline
        team: team-1
        config_file: path/to/config/file
        vars_files:
        - path/to/optional/vars/file/1
        - path/to/optional/vars/file/2
        vars:
          my_var: "foo"
          my_complex_var: {abc: 123}
  • pipelines: Required. Array of pipelines to configure. Must be non-nil and non-empty. The structure of the pipeline object is as follows:
  • name: Required. Name of pipeline to be configured. Equivalent of -p my-pipeline-name in fly set-pipeline command.

  • team: Required. Name of the team to which the pipeline belongs. Equivalent of -n my-team in fly login command. Must match one of the teams provided in source.

  • config_file: Required. Location of config file. Equivalent of -c some-config-file.yml in fly set-pipeline command.

  • vars_files: Optional. Array of strings corresponding to files containing variables to be interpolated via {{ }} in config_file. Equivalent of -l some-vars-file.yml in fly set-pipeline command.

  • vars: Optional. Map of keys and values corresponding to variables to be interpolated via (( )) in config_file. Values can arbitrary YAML types. Equivalent of -y "foo=bar" in fly set-pipeline command.

  • unpaused: Optional. Boolean specifying if the pipeline should be unpaused after the creation. If it is set to true, the command unpause-pipeline will be executed for the specific pipeline.

  • exposed: Optional. Boolean specifying if the pipeline should be exposed after the creation. If it is set to true, the command expose-pipeline will be executed for the specific pipeline.

dynamic

Resource configuration as above for Check, with the following job configuration:

---
jobs:
- name: set-my-pipelines
  plan:
  - put: my-pipelines
    params:
      pipelines_file: path/to/pipelines/file
  • pipelines_file: Required. Path to dynamic configuration file. The contents of this file should have the same structure as the static configuration above, but in a file.

Developing

Prerequisites

  • golang is required - version 1.13.x is tested; earlier versions may also work.
  • docker is required - version 17.06.x is tested; earlier versions may also work.

Dependencies

Dependencies are handled using go modules.

Updating dependencies

go mod download

To add or update a specific dependency version, follow the go modules instructions for Daily Workflow

Running the tests

Using a local environment

The acceptance tests require a running Concourse configured with basic auth to test against.

Run the tests with the following command (optionally also setting INSECURE=true):

FLY_LOCATION=path/to/fly \
TARGET=https://my-concourse.com \
USERNAME=my-basic-auth-user \
PASSWORD=my-basic-auth-password \
./bin/test

Using a Dockerfile

Note: the Dockerfile tests do not run the acceptance tests, but ensure a consistent environment across any docker enabled platform. When the docker image builds, the tests run inside the docker container, and on failure they will stop the build.

The tests need to be ran from one directory up from the directory of the repo. They will also need the fly linux tarball (from https://github.com/concourse/concourse/releases) to be present in the fly/ folder e.g:

$cwd/
├── fly/
│   └── fly-5.0.0-linux-amd64.tgz
└── concourse-pipeline-resource/
    ├── .git/
    │    └── ... 
    ├── dockerfiles/
    │    ├── alpine/
    │    │    └── Dockerfile
    │    └── ubuntu/
    │         └── Dockerfile
    └── ...

Run the tests with the following commands for both alpine and ubuntu images:

docker build -t concourse-pipeline-resource -f concourse-pipeline-resource/dockerfiles/alpine/Dockerfile .
docker build -t concourse-pipeline-resource -f concourse-pipeline-resource/dockerfiles/ubuntu/Dockerfile .

Contributing

Please ensure the tests pass locally.

concourse-pipeline-resource's People

Contributors

andrewedstrom avatar arbourd avatar chenbh avatar cirocosta avatar cnelson avatar eedwards-sk avatar jtarchie avatar jtxyz avatar mboldt avatar mhuangpivotal avatar novas0x2a avatar o-orand avatar onichols-pivotal avatar peterellisjones avatar pivotal-bin-ju avatar robdimsdale avatar trobert2 avatar vito avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

concourse-pipeline-resource's Issues

Bump fly to version 4.2.1

With the release of 4.2.1 we've upgraded our cluster but the resource is still behind (4.1.0).
We'll manage for now without the resource but any idea on when it might get bumped?

Pipeline resource fly version must match Concourse deployment

Since the concourse-pipeline-resource image contains the fly binary and does not fly sync, this leads to a failure of the resource when either the resource's fly version is ahead of the Concourse deployment or whenever the Concourse deployment is ahead of the concourse-pipeline-resource.

The workaround for this is to use a specific version of the resource by specifying the docker tag, but then the pipeline configuration is required to be changed just because the Concourse version is upgraded.

It would be better if the resource did a fly sync internally to be sure that it is using the correct version of fly for the Concourse deployment. Even better if the resource could just have a number of versions of fly internally & call an api to determine which it should use.

Latest docker image only works for Concourse 6.1

Today we started to see an error with this output when puting to our concourse-pipeline resource:

Logging to /tmp/concourse-pipeline-resource-out.log671813192
2020/05/14 23:40:05 exit status 1 - WARNING:

fly version (6.1.0) is out of sync with the target (5.8.1). to sync up, run the following:

    /opt/resource/fly -t https://ci.prs.releng.rocks sync

error: oauth2: cannot fetch token: 401 Unauthorized
Response: {"error":"invalid_client","error_description":"Invalid client credentials."}

When we dug into the problem we noticed that the fly version (6.1.0) that now ships in the latest Docker image exits with a 1 when fly login is called and the CLI version is out of sync. Previous versions of fly login warned in this situation, but exited 0. This change breaks the logic in fly.Command.Login(), which assumes that login will succeed even if the CLI is out of sync, and it can THEN sync it.

We're still on Concourse 5.8.1, but I imagine this same problem will happen for people on Concourse 6.0.* or (in the future) 6.2.*. For now we've switched from the latest tag of the Docker image to 2.2.0 to avoid the problem.

If you feel this is an unintended breaking change in the fly CLI, we can instead file an issue there.

'latest' concourse-pipeline-resource Docker image cannot log into Concourse v3.14.1

The latest docker image (updated at around 15:00 UTC 8th August) cannot log into Concourse v3.14.1, showing the following error:

Logging to /tmp/concourse-pipeline-resource-out.log013250056
2018/08/08 15:04:15 exit status 1 - not authorized. run the following to log in:

    fly -t https://***concourse-url*** login

Can the docker images please use appropriate tags, so we can use the versions of the resource appropriate for our version of concourse?

Fly version mismatch

Hi

Two of my pipelines (in my 5.8.0 Concourse deployment) use this resource, and they suddenly started failing their checks this morning with

fly version (6.1.0) is out of sync with the target (5.8.0). to sync up, run the following:

At the time of writing, the last successful runs were 22 hours ago, and the new version of this resource arrived on Dockerhub 21 hours ago, which suggests that something's broken in that release. Has anybody else been hit by this and maybe found a workaround?

Thanks

Sam

Concourse API client assumes a "target" url that is authenticated

It looks like the target URL that is used in the concourse API client assumes an authenticated URL (meaning basic auth credentials embedded into the URL). What this means is that a user will have to provide resource configuration like the following to make this resource work:

resources:
- name: some-pipeline
  type: concourse-pipeline
  source:
    target: https://username:[email protected]
    username: username
    password: password

We found this when we tried to use the resource against a concourse that is protected by basic auth. We were unable to do a put to create a new pipeline. We saw output like the following:

Logging to /tmp/concourse-pipeline-resource-out.log789647078
2016/05/10 00:06:57 Unexpected response status code: 401, expected: 200

And when we hijacked the put container, we saw this in the logs:

/tmp/build/put # more /tmp/concourse-pipeline-resource-out.log027887453
Concourse Pipeline Resource version: 0.2.2
Received input: {Source:{Target:https://my-concourse.com Username:username Password:***REDACTED-PASSWORD***} Params:{Pipelines:[{Name:pipeline ConfigFile:pipeline/pipeline.yml VarsFiles:[]}] PipelinesFile:}}
Performing login
Starting fly command: [-t concourse-pipeline-resource-target login -c https://my-concourse.com -u username -p ***REDACTED-PASSWORD***]
Waiting for fly command: [-t concourse-pipeline-resource-target login -c https://my-concourse.com -u username -p ***REDACTED-PASSWORD***]
Login successful
Input pipelines: [{Name:pipeline ConfigFile:pipeline/pipeline.yml VarsFiles:[]}]
Setting pipelines
Starting fly command: [-t concourse-pipeline-resource-target set-pipeline -n -p pipeline -c /tmp/build/put/pipeline/pipeline.yml]
Waiting for fly command: [-t concourse-pipeline-resource-target set-pipeline -n -p pipeline -c /tmp/build/put/pipeline/pipeline.yml]
Setting pipelines complete
Getting pipelines
Exiting with error: Unexpected response status code: 401, expected: 200

We think the issue is that the URL given to this GET request (https://github.com/robdimsdale/concourse-pipeline-resource/blob/14e4a76d1470c336047d9c912a37f6d406b0416c/concourse/api/pipelines.go#L31) does not include the basic auth credentials if you don't include them with the target like is shown above.

fly sync issue

After the changes introduced to use fly cli 6.1.0, concourse-pipeline resource fails during put operation with error

Logging to /tmp/concourse-pipeline-resource-out.log411759429
2020/05/14 18:26:48 exit status 1 - WARNING:

fly version (6.1.0) is out of sync with the target (5.8.1). to sync up, run the following:

    /opt/resource/fly -t PE sync

error: oauth2: cannot fetch token: 401 Unauthorized
Response: {"error":"invalid_client","error_description":"Invalid client credentials."}```

Support reading team from Vault

Tried to read the team name from Vault. Getting this error:
2018/02/07 10:11:39 team name '((concourse-team))' not found in source team names:

However, I am able to read both the user and password from vault.

Not available for 3.13.0

I've upgraded to 3.13.0 a few days ago and am now receiving

fly version (3.12.0) is out of sync with the target (3.13.0). to sync up, run the following

errors from my concourse-pipeline resources. I think you haven't yet released a new version of the resource since the 3.13 update.

Can you release a new version of the pipeline? I think you just need to trigger a build.

Doesn't work with 3.3.0

the check gets the following error:

json: unsupported type: map[interface {}]interface {}

cc @chendrix the necessary changes to the codebase to support the periodic breaking changes in the API might be good justification to prioritize a switch to fly

fly version (4.0.0) is out of sync with the target (4.2.1). to sync up

I have tried to change the resource type of my pipeline-resource to get it past this error message. I have used 4.0.0, latest, and v4.0-pre. Is there a fix to this.

  • Concourse CI v4.2.1 for all (Concourse and Fly)
  • Docker version 18.06.1-ce, build e68fc7a
  • docker-compose version 1.18.0, build 8dd22a9
  • VERSION="16.04.5 LTS (Xenial Xerus)"
- name: concourse-pipeline
  type: docker-image
  source:
    repository: redacted_website./concourse/concourse/concourse-pipeline-resource
    tag: v4.0-pre
    username: ((artifactory-username))
    password: ((artifactory-password))
Logging to /tmp/concourse-pipeline-resource-out.log048495527
2018/10/12 18:58:01 exit status 1 - WARNING:
fly version (4.0.0) is out of sync with the target (4.2.1). to sync up, run the following:	
fly -t https://IP_ADDRESS:8443 sync
error: oauth2: cannot fetch token: 401 Unauthorized	
Response: {"error":"access_denied","error_description":"Invalid username or password"}

Hide stdout from Fly / Hide secrets?

Scenario
We have secrets in our pipelines imported from variables / vault that make it through to the fly output via the set-pipeline. Using this resource it will have an audit of these secrets, which would preferably be hidden or just not exist completely.

Implementation

  1. Hide the stdout entirely via a new out parameter.
  2. Consider certain vars "secrets" to prevent these specific variables being visible in the output (this is more a fly specific and therefore probably completely out of scope for this issue).

I believe 1 is the best scenario and easiest to cover. If the stdout is only in the interim containers, which get cleaned up, I think this would be sufficient for our use case.

Is this something we would want added to this resource?

Property Does Not Exist: Concourse:Resource-Result

Deployed via concourse helm chart, added a pipeline to set a pipeline from a yaml in a mount point, but the worker is throwing the following for the job:

{"timestamp":"2019-06-04T06:44:52.891945787Z","level":"error","source":"guardian","message":"guardian.api.garden-server.get-property.failed","data":{"error":"property does not exist: concourse:resource-result","handle":"8af14252-4e9a-4264-7cb9-d916ee0f59ff","session":"3.1.236"}}

Pipeline after pulling states the following:

Logging to /tmp/concourse-pipeline-resource-out.log25509096

I can't see any other details on this. Can't find the log in the worker either.

This is the file I used:


resource_types:
- name: concourse-pipeline
  type: docker-image
  source:
    repository: concourse/concourse-pipeline-resource
    tag: 2.1.1

resources:
- name: my-pipelines
  type: concourse-pipeline
  source:
    teams:
    - name: main

jobs:
- name: set-my-pipelines
  plan:
  - put: my-pipelines
    params:
      pipelines:
      - name: alertd
        team: main
        config_file: /pipelines/pipeline.yaml

[Bug] Resource does not allow for an empty list of pipelines to be put

For the following config:

---
jobs:
- name: set-my-pipelines
  plan:
  - put: my-pipelines
    params:
      pipelines_file: path/to/pipelines/file.yaml

where the contents of path/to/pipelines/file.yaml are:

---
pipelines: []

Expected behavior: The put step silently succeeds, having done nothing because it was configured to do nothing.

Actual behavior: The put step fails with the following error: pipelines must be provided via either pipelines or pipelines_file

Context / use-case: we have a meta-pipeline which dynamically generates pipeline configurations for a variety of Concourse targets. Sometimes, only some of the Concourse targets have configurations generated for them, leaving the resources for the other Concourse targets with nothing to put. When this happens, it is intentional, and the build should not be failed.

Is there an image for fly 5.x.x?

The lastest build cannot be used with the version 5.x.x of concourse. Did you keep an image for backward compatibility in see 2.0.0, 3.0.0 and 5.0.0 on dockerhub but no 5.0.0

[bug] build fails on macOS

Build as described in documentation fails.

$ git clone https://github.com/concourse/concourse-pipeline-resource.git
$ cd concourse-pipeline-resource
$ docker build -t concourse-pipeline-resource .
Sending build context to Docker daemon  9.686MB
Step 1/18 : FROM golang:alpine as builder
 ---> 85256d3905e2
Step 2/18 : RUN apk add --no-cache curl jq
 ---> Using cache
 ---> f737adc68714
Step 3/18 : RUN mkdir -p /assets
 ---> Using cache
 ---> 5aa762b7a0bc
Step 4/18 : RUN url=$(curl -s "https://api.github.com/repos/concourse/concourse/releases/latest"     | jq -r '.assets[] | select(.name | test("fly_linux_amd64")) | .browser_download_url') &&    curl -L "$url" -o /assets/fly
 ---> Using cache
 ---> 9f8b8f918749
Step 5/18 : COPY . /go/src/github.com/concourse/concourse-pipeline-resource
 ---> 86993cd443d2
Step 6/18 : ENV CGO_ENABLED 0
 ---> Running in 2dfa053cd06b
Removing intermediate container 2dfa053cd06b
 ---> d61314811016
Step 7/18 : RUN go build -o /assets/in github.com/concourse/concourse-pipeline-resource/cmd/in
 ---> Running in 9ceb9ed12dbb
Removing intermediate container 9ceb9ed12dbb
 ---> 544c3fbd8a61
Step 8/18 : RUN go build -o /assets/out github.com/concourse/concourse-pipeline-resource/cmd/out
 ---> Running in 453613cfbf16
Removing intermediate container 453613cfbf16
 ---> 14f38727ec72
Step 9/18 : RUN go build -o /assets/check github.com/concourse/concourse-pipeline-resource/cmd/check
 ---> Running in fbd96dac63d0
Removing intermediate container fbd96dac63d0
 ---> da0070214e39
Step 10/18 : RUN set -e; for pkg in $(go list ./... | grep -v "acceptance"); do 		go test -o "/tests/$(basename $pkg).test" -c $pkg; 	done
 ---> Running in 9f12290257aa
# github.com/concourse/concourse-pipeline-resource/cmd/check
src/github.com/concourse/concourse-pipeline-resource/cmd/check/main.go:75: Fatalln call has possible formatting directive %v
The command '/bin/sh -c set -e; for pkg in $(go list ./... | grep -v "acceptance"); do 		go test -o "/tests/$(basename $pkg).test" -c $pkg; 	done' returned a non-zero code: 2

Can't use docker network DNS resolution for target

We're using the docker images (on ECS) and the worker container is linked to the web container on the address concourse-web. ECS/Docker modifies /etc/hosts but presumably that isn't propagated through to the garden resource check containers. Is it possible to verify whether this resource would actually look at /etc/hosts?

Thanks.

Resource cache error with too many pipelines

What happened: Updating pipelines failed with error: pq: index row size 2760 exceeds maximum 2712 for index "resource_caches_resource_config_id_version_params_hash_key"

Reason: The version string generated by the pipeline resource is too long. It is a map of pipeline-name->md5(pipeline-content)

The version string is used as part of an index in the resource_caches table. https://github.com/concourse/atc/blob/176c130dc78f77eb0350b42c8b40312a3cbf3547/db/migrations/124_create_caches.go#L77

With a lot of pipelines you can exceed the maximum index row size for postgres, and will hit this problem.

Suggesting to change the version string to something that can't increase without an upper bound. Maybe have it become the sha256 of the current version string.

Very vague error in using this resource..

We are trying to use this to set a pipeline and get an error that doesn't describe a whole lot... Is there anyway to debug this?

Logging to /tmp/concourse-pipeline-resource-out.log922228100
2018/10/15 17:37:29 exit status 1 - error: resource not found

Bump fly version to 5.60

This resources gives the following error when running against Concourse 5.60

Logging to /tmp/concourse-pipeline-resource-out.log964080286
2019/10/24 10:37:59 exit status 1 - WARNING:

fly version (5.3.0) is out of sync with the target (5.6.0). to sync up, run the following:

    fly -t http://example.com sync

error: oauth2: cannot fetch token: 400 Bad Request
Response: {"error":"invalid_request","error_description":"Requested connector does not exist."}

Could fly be updated to 5.6.0?

Support for destroying pipelines

We should ultimately add delete to the resource interface for this, but until then it'd be nice if put supported destroying a set of pipelines. Probably from a file listing their names.

The main use case for this is automating Concourse pipeline lifecycles based on Git branches via https://github.com/vito/git-branches-resource - which detects new and removed branches.

[bug] the resource gets from Concourse ALL the existing pipelines instead of only the ones mentioned in the `put`

Bug Report

The resource gets from Concourse all the existing pipelines, instead of focusing only on the ones managed by it (that is, the ones that are put).

In my understanding this has two problems:

  • when the ATC has lots of pipelines, this makes the check/get/put take more time than needed. This indirectly already caused some failures as detailed in #47.
  • all these unrelated pipelines are printed in the UI both under the put and under the explicit get that comes after, making very confusing to understand what is going on.

Feature request to support input variables

Currently you can specify a configuration file for input variables, like the fly option --load-vars-from= ; but it would be good to input variables directly without needing a separate configuration file, along the lines of the fly command --var=[SECRET=KEY].

Suggested syntax could be:


---
jobs:
- name: set-my-pipelines
  plan:
  - put: my-pipelines
    params:
      pipelines:
      - name: my-pipeline
        config_file: path/to/config/file
        vars_files:
        - path/to/optional/vars/file/1
        vars:
        - some_variable: true
        - secret: key

Please provide older versions of fly

Some of us can't upgrade to the latest version as soon as it arrives.

It would be very much appreciated if you can make the resource docker image(s) available, to pull from Docker Hub, for older fly versions.

Perfectly correct pipeline is rejected "resource not used"

Hi,

How do you check for resource not used. I have resource that is used only in on_failier, but still used. When I try to create pipeline with this resource it is rejected. Note that the same pipeline is accepted from concourse 2.7.0 directly.

[feature] flag to unpause pipelines

It would be nice to have a way to unpause a pipeline after creating it. The lack of this feature means that we have to create one more fly resource and execute that command. Since we do that, there is no reason to use this resource.

I propose having a flag called unpause in the configuration of each pipeline with a default to false:

      pipelines:
      - name: my-pipeline
        team: team-1
        config_file: path/to/config/file
        unpause: true
        vars_files:
        - path/to/optional/vars/file/1
     - name: my-second-pipeline
        team: team-2
        config_file: path/to/config/file2
        vars_files:
        - path/to/optional/vars/file/2

failure after concourse 5.0 upgrade

Noticed the following errors for concourse-pipeline-resource after upgrading to concourse 5.0, is it something related to RBAC changes?

resource script '/opt/resource/check []' failed: exit status 1

stderr:
Logging to /tmp/concourse-pipeline-resource-check.log007011057
2019/03/07 04:20:34 exit status 1 - WARNING:

fly version (4.2.2) is out of sync with the target (5.0.0). to sync up, run the following:

    fly -t https://my.concourse.net sync

error: oauth2: cannot fetch token: 400 Bad Request
Response: 

Pipeline not found error at put

We use concourse-pipeline-resource with concourse 2.7.1 without any issues.

Since the update of concourse to 3.1.1, concourse-pipeline-resource fail (re-)creating pipelines which doesn't already exist.

The error message is:
2017/06/14 13:44:42 error from target: http://<concourse_url> - Pipeline not found: <pipeline_name>

Best Regards

Doesn't work on Concourse 6.5

We've upgraded our Concourse setup to version 6.5.1, but now our pipeline generators stopped working because the used fly version in this resource is not compatible with the new authentication changes.

I know that this resource is being deprecated, but we haven't been able to move to the new set_pipeline method because of the limitation of not working across teams.

Can you ship a new version of the resource with an updated version of fly?

check returns a different version even if pipeline config didn't change

For every run of check, there will be one pipeline version that changes because the fly get-pipeline command will return the pipeline's config and the resources will be sorted differently thus varying the resulting md5sum. This will then unnecessarily record a new version of the resource in the versioned_resources db table.

Feature to expose pipelines

Might be handy to have a feature to run expose-pipeline when creating pipelines for teams.
E.g adding a public: true setting.

- put: my-pipelines
    params:
      pipelines:
      - name: my-pipeline
        team: team-1
        public: true
        config_file: path/to/config/file
        vars_files:
        - path/to/optional/vars/file/1
        - path/to/optional/vars/file/2

Fails in 6.1.0

With reference to #71 I am now testing Concourse 6.1.0. I have removed the tag so I'm using the latest version of this resource, but now in the check I get:

run check step: run check step: check: resource script '/opt/resource/check []' failed: exit status 1

stderr:
Logging to /tmp/concourse-pipeline-resource-check.log692886880
2020/05/21 15:06:04 exit status 1 - error: resource not found

Am I doing something dumb?

Having a public build available

It could be great to have a public build available for this resource (including the docker image build). I've setup one on circleci. If you're interested, I can PR the config file. You can have a look here: master branch build. I know it's not concourse and it seems you're already using a private concourse, but there isn't any public concourse I'm aware of.

Convert insecure to boolean

Hello,
Back in 2016, concourse only supported string variable (to be able to set it using a var file), but now since 3.2.0 we have strings, boolean values, arrays. So rather than putting string everywhere, it could be better to convert insecure to boolean ? But it is going to break compatibility with previous version.
What do you think ?

Login OAuth

Does the concourse-pipeline-resource support authenticating with a Concourse install with GitHub OAuth? I can't see it mentioned in the README, only basic auth.

Not deploying minor version change as latest version

Not deploying the latest version from artifactory after successfully fetching the latest version

Bug Report

Pipeline is unable to detect and deploy minor version change as latest version from artifactory. This is due to version number increment from X.X.99 > X.X.100 > X.X.101

Steps to Reproduce

After pipeline successfully run tests, version is up from X.X.99 to X.X.100. Pipeline deploys version X.X.99 on server and showing X.X.99 as latest version on pipeline while X.X.100 is placed at the bottom of the list of version as an older version. I had to disable every old version (X.X.99...) and only enabling the latest version (X.X.100) to deploy the latest version.

Note: Screen capture illustrates the issue with X.X.101 instead of X.X.100
concourse version issue-1-edited
concourse version issue-2-edited

Expected Results

It should detect X.X.100 as the latest version and place it right at the top instead of X.X.99 as latest and placed X.X.100 as old version

Actual Results

Pipeline still deploys version X.X.99 on server and showing X.X.99 as latest version on pipeline while X.X.100 is placed at the bottom of the list of version as an older version.

Additional Context

N.A.

Version Info

  • Concourse version: v5.3.0
  • Deployment type (BOSH/Docker/binary): BOSH
  • Infrastructure/IaaS: -
  • Browser (if applicable): Chrome 76
  • Did this used to work? No

Use fly instead of go-concourse

Main issue: concourse/concourse#1225

We want to use the concourse-pipeline-resource in the production pipelines for Concourse. In order to do this, we need to strip out all the code that is making calls to go-concourse and replace it with code that shells out to fly. go-concourse is an undocumented api that we use internally in the fly command and is not safe to develop against. It has changed since this repo was last updated, breaking the pipeline resource for more recent versions of Concourse (See issues #17 and #13 for examples of things that this has broken).

We do make guarantees about Fly's API, so it will not up and change in the same way that go-concourse will. Fly is safer to develop against, which is why we're reworking this to use fly.

This work is done now and we are back to parity using fly instead of go-concourse. Now we are just backfilling tests, adding automated fly syncing, and adding jobs to the resources pipeline on ci.concourse.ci to automatically build the docker image for this resource.

Authentication missing since max 3 days

Hi,
we wanted to execute our pipeline that uses this resource today.

But we get the following error:
exit status 1 - not authorized. run the following to log in: fly -t https://url login

the last time we executed the pipeline was 3 days ago and everything was fine.
now i see there was an update yesterday. As i cannot go back to an older version of that resource, is it possible that the changes made yesterday can lead to this problem?

We did not change anything else, so this is my last hope/idea

I hope you can help

maybe its related to using another fly? c685998

I also find errors like:
Exiting with error: exit status 1 - error: unknown flag json'`

Some more logs:

Setting pipelines
Performing login
Starting fly command : login
Waiting for fly command: login
Starting fly command : sync
Waiting for fly command: sync
Login successful
Starting fly command : set-pipeline
Waiting for fly command: set-pipeline
Exiting with error: exit status 1 - not authorized. run the following to log in:

I took out every specific stuff. I check all arguments and they are correct.
On the first view it looks like the login was not saved somehow..

Support auth with oauth client credentials

Now that fly supports auth via oauth token, this resource should be able to authenticate with an oauth token endpoint and client credentials as an alternative to basic auth. If you all are interested in this feature, I'd be happy to work up a patch.

cc @chendrix

This resource will not work with concourse 6.1.0

As already mentioned in #65 this resource uses a pinnend fly version of 5.3.0 which is outdated but was somehow compatible until concourse 6.0.0. With the latest release of concourse 6.1.0 which introduced a breaking change in the authentication this resource will now stop working.

@vito will there be support for this resource in the future? if not it would maybe be helpful to add a disclaimer to the readme.

could not reach the Concourse server

After successfully putting a pipeline the resources fails the implicit getting of the pipeline with the following error. Looking at the log file it appears that the resources is opening parallel connections to concourse to fetch all the pipelines and some of those connections are timing out. We have a few dozen pipelines in the same concourse deployment. This issue didn't appear until we added more than a dozen or so pipelines.

Logging to /tmp/concourse-pipeline-resource-in.log208407758
2018/11/09 23:44:30 exit status 1 - could not reach the Concourse server called https://concourse.apachegeode-ci.info:

    Get https://concourse.apachegeode-ci.info/api/v1/info: dial tcp: i/o timeout

is the targeted Concourse running? better go catch it lol

Is there a reason why the get needs to download all the pipelines? Is there an easy way to increase the timeout or retry on timeout? Can it be done serially like it appears to do on the put? Any help would be great.

Thanks!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.