Giter VIP home page Giter VIP logo

semver-resource's Introduction

Semver Resource

A resource for managing a version number. Persists the version number in one of several backing stores.

Source Configuration

  • initial_version: Optional. The version number to use when bootstrapping, i.e. when there is not a version number present in the source.

  • driver: Optional. Default s3. The driver to use for tracking the version. Determines where the version is stored.

There are four supported drivers, with their own sets of properties for configuring them.

git Driver

The git driver works by modifying a file in a repository with every bump. The git driver has the advantage of being able to do atomic updates.

  • uri: Required. The repository URL.

  • branch: Required. The branch the file lives on.

  • file: Required. The name of the file in the repository.

  • private_key: Optional. The SSH private key to use when pulling from/pushing to to the repository.

  • username: Optional. Username for HTTP(S) auth when pulling/pushing. This is needed when only HTTP/HTTPS protocol for git is available (which does not support private key auth) and auth is required.

  • password: Optional. Password for HTTP(S) auth when pulling/pushing.

  • git_user: Optional. The git identity to use when pushing to the repository support RFC 5322 address of the form "Gogh Fir <[email protected]>" or "[email protected]".

  • depth: Optional. If a positive integer is given, shallow clone the repository using the --depth option.

  • skip_ssl_verification: Optional. Skip SSL verification for git endpoint. Useful for git compatible providers using self-signed SSL certificates.

  • commit_message: Optional. If specified overides the default commit message with the one provided. The user can use %version% and %file% to get them replaced automatically with the correct values.

s3 Driver

The s3 driver works by modifying a file in an S3 compatible bucket.

  • bucket: Required. The name of the bucket.

  • key: Required. The key to use for the object in the bucket tracking the version.

  • access_key_id: Required. The AWS access key to use when accessing the bucket.

  • secret_access_key: Required. The AWS secret key to use when accessing the bucket.

  • assume_role_arn: Optional. The AWS role to assume when using access keys.

  • session_token: Optional. The AWS session token to use when accessing the bucket.

  • region_name: Optional. Default us-east-1. The region the bucket is in.

  • endpoint: Optional. Custom endpoint for using S3 compatible provider.

  • disable_ssl: Optional. Disable SSL for the endpoint, useful for S3 compatible providers without SSL.

  • skip_ssl_verification: Optional. Skip SSL verification for S3 endpoint. Useful for S3 compatible providers using self-signed SSL certificates.

  • server_side_encryption: Optional. The server-side encryption algorithm used when storing the version object (e.g. AES256, aws:kms).

  • use_v2_signing: Optional. Use v2 signing, default false.

swift Driver

The swift driver works by modifying a file in a container.

  • openstack Required. All openstack configuration must go under this key.

    • container: Required. The name of the container.

    • item_name: Required. The item name to use for the object in the container tracking the version.

    • region: Required. The region the container is in.

    • identity_endpoint, username, user_id, password, api_key, domain_id, domain_name, tenant_id, tenant_name, allow_reauth, token_id: See below The swift driver uses gophercloud to handle interacting with OpenStack. All OpenStack Identity versions are supported through this library. The Authentication properties will pass through to it. For detailed information about the individual parameters, see https://github.com/rackspace/gophercloud/blob/master/auth_options.go

gcs Driver

The gcs driver works by modifying a file in a Google Cloud Storage bucket.

  • bucket: Required. The name of the bucket.

  • key: Required. The key to use for the object in the bucket tracking the version.

  • json_key: Required. The contents of your GCP Account JSON Key. Example:

    json_key: |
      {
        "private_key_id": "...",
        "private_key": "...",
        "client_email": "...",
        "client_id": "...",
        "type": "service_account"
      }

Example

With the following resource configuration:

resources:
- name: version
  type: semver
  source:
    driver: git
    uri: [email protected]:concourse/concourse.git
    branch: version
    file: version
    private_key: {{concourse-repo-private-key}}

Bumping with a get and then a put:

plan:
- get: version
  params: {bump: minor}
- task: a-thing-that-needs-a-version
- put: version
  params: {file: version/version}

Or, bumping with an atomic put:

plan:
- put: version
  params: {bump: minor}
- task: a-thing-that-needs-a-version

Behavior

check: Report the current version number.

Detects new versions by reading the file from the specified source. If the file is empty, it returns the initial_version. If the file is not empty, it returns the version specified in the file.

in: Provide the version as a file, optionally bumping it.

Provides the version number to the build as a version file in the destination.

Can be configured to bump the version locally, which can be useful for getting the final version ahead of time when building artifacts.

Parameters

Note that bump and pre don't update the version resource - they just modify the version that gets provided to the build. An output must be explicitly specified to actually update the version.

  • pre_without_version: Optional. By default false, once it's set to true then PreRelease will be bumped without a version number.

out: Set the version or bump the current one.

Given a file, use its contents to update the version. Or, given a bump strategy, bump whatever the current version is. If there is no current version, the bump will be based on initial_version.

The file parameter should be used if you have a particular version that you want to force the current version to be. This can be used in combination with in, but it's probably better to use the bump and pre params as they'll perform an atomic in-place bump if possible with the driver.

Parameters

One of the following must be specified:

  • file: Optional. Path to a file containing the version number to set.

  • bump and pre: Optional. See Version Bumping Semantics.

When bump and/or pre are used, the version bump will be applied atomically, if the driver supports it. That is, if we pull down version N, and bump to N+1, the driver can then compare-and-swap. If the compare-and-swap fails because there's some new version M, the driver will re-apply the bump to get M+1, and try again (in a loop).

  • pre_without_version: Optional. By default false, once it's set to true then PreRelease will be bumped without a version number.

  • get_latest: Optional. See Check-less Usage.

Version Bumping Semantics

Both in and out support bumping the version semantically via two params: bump and pre:

  • bump: Optional. Bump the version number semantically. The value must be one of:

    • major: Bump the major version number, e.g. 1.0.0 -> 2.0.0.
    • minor: Bump the minor version number, e.g. 0.1.0 -> 0.2.0.
    • patch: Bump the patch version number, e.g. 0.0.1 -> 0.0.2.
    • final: Promote the version to a final version, e.g. 1.0.0-rc.1 -> 1.0.0.
  • pre: Optional. When bumping, bump to a prerelease (e.g. rc or alpha), or bump an existing prerelease.

    If present, and the version is already a prerelease matching this value, its number is bumped. If the version is already a prerelease of another type, (e.g. alpha vs. beta), the type is switched and the prerelease version is reset to 1. If the version is not already a pre-release, then pre is added, starting at 1.

    The value of pre can be anything you like; the value will be pre-pended (hah) to a numeric value. For example, pre: foo will result in a semver of x.y.z-foo.<number>, pre: alpha becomes x.y.z-alpha.<number>, and pre: my-preferred-naming-convention becomes x.y.z-my-preferred-naming-convention.<number>

  • build: Optional. Same as pre but for build labels (e.g. build: foo will result in a semver of x.y.z+foo.<number>, build: alpha becomes x.y.z+alpha.<number>.

    It is valid for a semver to be both a prerelease and a build, for example, pre: alpha, build: test results in x.y.z-alpha.<number>+test.<number>

  • pre_without_version: Optional. When bumping to a prerelease, drop the version if set to true. Examples:

    • Major version bump: version file = 1.2.4-SNAPSHOT, release version = 2.0.0
    • Minor version bump: version file = 1.2.4-SNAPSHOT, release version = 1.3.0
    • Promote snapshot: version file = 1.2.4-SNAPSHOT, release version = 1.2.4
  • build_without_version: Optional. Same as pre_without_version but for build labels.

Check-less Usage

A classic usage of semver resource is like:

resources:
- name: version
  type: semver
  source:
    driver: git
    uri: [email protected]:concourse/concourse.git
    branch: version
    file: version
    private_key: {{concourse-repo-private-key}}

jobs:
- name: some-job
  plan:
  - get: trigger-resource
    trigger: true
  - get: version
    param: {bump: major}
  - task: a-thing-that-needs-a-version
  - put: version
    params: {file: version/version}

In above classic mode, Concourse will run periodic checks against the semver resource version. Each check will do a git clone as the driver is git. When there are a lot of semver resources, checks on semver resources may also bring burden to the git system as each check will invoke a git clone.

Given each semver resource requires a parameter file in source, semver resources are hard to enjoy benefits of global resources.

To mitigate the burden of checks, if a semver resource is not a job trigger, check-less mode can be used. The above sample then can be rewritten as:

jobs:
- name: some-job
  plan:
  - get: trigger-resource
    trigger: true
  - put: version # change `get` to `put`
    params:
      get_latest: true # and set `get_latest: true`
      bump: major
  - task: a-thing-that-needs-a-version
  - put: version
    params: {file: version/version}

You may have noticed that, original get: version is changed to put: version. Now resource version is put-only, then Concourse will no longer run check on it. Param get_latest: true tells the put step to only fetch the latest version without bumping anything. Then the implied get will fetch a version as a typical get step.

If your Concourse or Git (e.g. Gitlab) systems are exhausted by semver resources' checks, you may consider reforming pipelines to use this check-less usage.

The cons of check-less usage are:

  • you cannot use put step as a job trigger.
  • put step with get_latest: true will always fetch the latest version, thus you are not able to pin an old version.

Running the tests

The tests have been embedded with the Dockerfile; ensuring that the testing environment is consistent across any docker enabled platform. When the docker image builds, the test are run inside the docker container, on failure they will stop the build.

Run the tests with the following command:

docker build -t semver-resource --target tests --build-arg base_image=paketobuildpacks/run-jammy-base:latest .

Integration tests

The integration requires two AWS S3 buckets, one without versioning and another with. The docker build step requires setting --build-args so the integration will run.

You will need:

  • AWS key and secret
  • An S3 bucket
  • The region you are in (i.e. us-east-1, us-west-2)

Run the tests with the following command, replacing each build-arg value with your own values:

docker build . -t semver-resource --target tests -f dockerfiles/alpine/Dockerfile \
  --build-arg SEMVER_TESTING_ACCESS_KEY_ID="some-key" \
  --build-arg SEMVER_TESTING_SECRET_ACCESS_KEY="some-secret" \
  --build-arg SEMVER_TESTING_BUCKET="some-bucket" \
  --build-arg SEMVER_TESTING_REGION="some-region"

docker build . -t semver-resource --target tests -f dockerfiles/ubuntu/Dockerfile \
  --build-arg SEMVER_TESTING_ACCESS_KEY_ID="some-key" \
  --build-arg SEMVER_TESTING_SECRET_ACCESS_KEY="some-secret" \
  --build-arg SEMVER_TESTING_BUCKET="some-bucket" \
  --build-arg SEMVER_TESTING_REGION="some-region"

Contributing

Please make all pull requests to the master branch and ensure tests pass locally.

semver-resource's People

Contributors

baszalmstra avatar bodin avatar cfryanr avatar chenbh avatar chendrix avatar dependabot[bot] avatar evanchaoli avatar evashort avatar fenech avatar glompix avatar goldstar611 avatar jamesclonk avatar jim80net avatar jlsalmon avatar jtarchie avatar leggomymeggos avatar lnguyen avatar mariash avatar mohsen0 avatar pvaramballypivot avatar samze avatar shinji62 avatar shyx0rmz avatar taylorsilva avatar theozaurus avatar vito avatar xenophex avatar xoebus avatar youssb avatar zrob avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

semver-resource's Issues

Initial_version is not optional in the Swift driver

The docs state that the initial_version parameter is optional. However if you are using the Swift driver and do not specify it explicitly you will get the following error:

Initial version was not a valid sem ver

version marked invalid still used and bumped incorrectly

Concourse 4.0.0

I marked a version as 'invalid', then proceeded to bump it.

The version it 'gets' is 0.0.0, yet it bumps it to 0.1.1 meaning it started from the 'bad' version. Seems like it's improperly using the cache or something.

- name: bump-patch
  serial_groups:
  - version
  plan:
  - get: version
  - put: version
    params:
      bump: patch

screen shot 2018-08-07 at 2 19 04 pm

screen shot 2018-08-07 at 2 19 28 pm

I thought maybe it was an 'atomic put' issue cause I was doing an unnecessary get, I think?

So I removed the - get: version line from bump-patch and tried marking the latest version as invalid again, and then 'bumping' it, and it still just immediately bumped to 0.1.2, again ignoring the fact that 0.1.1 was marked as 'bad' and the only remaining good version is 0.0.0

(I didn't grab a screenshot before it made 0.1.2, but you can see that 0.0.0 should have been the only valid version)

screen shot 2018-08-07 at 2 27 17 pm

Can't run tests without providing all build args

The README indicates that if there are no build args, all tests but the integration tests will run. However, running docker build -t semver-resource . gives the following error:

Failure [0.000 seconds]
[BeforeSuite] BeforeSuite
/go/src/github.com/concourse/semver-resource/check/check_suite_test.go:20

  must specify $SEMVER_TESTING_ACCESS_KEY_ID
  Expected
      <string>:
  not to be empty

  /go/src/github.com/concourse/semver-resource/check/check_suite_test.go:23

It seems like this shouldn't happen, or at least the README should be updated to remove the command without build args.

Output version to different directory

I have some tasks that span jobs in a pipeline that do exactly the same thing except for version number resources. What I'd really like to do is use the semver resource to output to a different directory so that I only need the one task file, and can have version/version as one input to it.

my setup is something like this:

task-requiring-version.yml

---
platform: linux

image_resource:
  type: docker-image
  source:
    repository: custom-docker-image

inputs:
  - name: pipelines
  - name: version
  
run:
  path: /bin/bash
  args: 
  - -cel
  - |
    # Do stuff requiring the version
    export version=$(cat ${version/version})
    # and so on

another-task-requiring-different-version.yml

---
platform: linux

image_resource:
  type: docker-image
  source:
    repository: custom-docker-image

inputs:
  - name: pipelines
  - name: different-version
  
run:
  path: /bin/bash
  args: 
  - -cel
  - |
    # Do stuff requiring the version
    export version=$(cat ${different-version/version})
    # and so on

pipeline.yml

- name: deploy app 1
  plan:
  - aggregate:
    - get: pipelines
    - get: version
      passed: ["previous step"] # where the version got bumped

  - task: Deploy
    file: pipelines/tasks/deploy-app-1.yml
  
  - task: Check application health
    file: pipelines/tasks/task-requiring-version.yml

- name: deploy app 2
  plan:
  - aggregate:
    - get: pipelines
    - get: different-version
      passed: ["previous other step"] # where the different-version got bumped

  - task: Deploy
    file: pipelines/tasks/deploy-app-2.yml
  
  - task: Check application health
    file: pipelines/tasks/another-task-requiring-different-version.yml.yml

# and so on

where as what I think I'd like is:

pipeline.yml

- name: deploy app 1
  plan:
  - aggregate:
    - get: pipelines
    - get: version
      passed: ["previous step"] # where the version got bumped

  - task: Deploy
    file: pipelines/tasks/deploy-app-1.yml
  
  - task: Check application health
    file: pipelines/tasks/task-requiring-version.yml

- name: deploy app 2
  plan:
  - aggregate:
    - get: pipelines
    - get: different-version
      params: {out-dir: version}
      passed: ["previous other step"] # where the different-version got bumped

  - task: Deploy
    file: pipelines/tasks/deploy-app-1.yml
  
  - task: Check application health
    file: pipelines/tasks/task-requiring-version.yml

# and so on

If there's already a known way of doing this that doesn't require another task to copy the versions to task outputs (which would require the same custom tasks, just doing something different), I'd appreciate being told how. Otherwise, does this sound like a feasible enhancement?

[Question] Combining semver with git resource

Hi,

I'd like to store version in Git repository. However, I'm falling into build loop if I'm not ignoring "version" file. If I do ignore "version" file, this leads to lost commits in future jobs (as they are ignored) and I'm unable to push back to repository. Perhaps someone has an idea how to solve this?

[Question] How to use specify a prerelease version as 1.0.0.build-1

We are evaluating moving our builds from 1.0.0.beta-1 to 1.0.0.build-1 similar to how Pivotal is versioning the tiles that they release on pivnet. We see a lot of benefits in this versioning scheme in that we would be able to build the artifact once (in this case a spring boot application) and promote it through the pipeline and not have to rebuild it when shiping it as a release. Instead we would move the preelease from the pre-release repository to the release repository and the binary will not change or the version. Is this feasible with the semver-resource in concourse? Is there a different resource that we might look at? Is this the same resource that Pivotal release engineering uses for versioning their tiles?

bump final and pre should guarantee an advancement of version

When you bump final a semver that's already final, I want the patch to be bumped. Currently, a new version is not produced.

Current:
1.0.0 --> bump final --> 1.0.0

Proposed:
1.0.0 --> bump final --> 1.0.1

When you bump pre a semver that's already final, I want the patch to be bumped, with the pre applied after. Currently, the pre is applied to the final semver, going backwards.

Current:
1.0.0 --> bump pre build --> 1.0.0-build.1

Proposed:
1.0.0 --> bump pre build --> 1.0.1-build.1

I'm using a script to to work around this currently, it would be nice to get away from that so I can have the above behaviour applied atomically.

Check binary tests fail due to new default format of keys generated via ssh-keygen

UAA team has forked from this repo and realized we recently started having some failures when running the test/check.sh tests during build of the Docker image. We have found that the same failure is exhibited in this repo as well.

Specifically, there is a test in test/check.sh called it_fails_if_key_has_password which attempts to run check given a private key encrypted with a passphrase. It appears the GitDriver attempts to inspect the contents of the private key to verify whether or not it is encrypted. Also, kind of related to this: there don't appear to be any unit tests for the GitDriver, don't know if we missed them or this was intentional?

Problem is that the newer generation of keys through ssh-keygen no longer contain headers indicating whether or not the key is encrypted. This invalidates the logic linked above by considering the key to be valid and being able to successfully fetch the version from the bucket when it shouldn't. (At least in that it_fails_if_key_has_password test)

We are looking into a fix for ourselves and can send one your way or hold off, up to y'all.

Thanks!

~ Jwal & @bruce-ricard

Git tag with smart tagging

If you have a long pipeline with a bunch of checks until you get to the point you would like to version it there might be new changes added to the git repo. When a git driver is used it is very misleading to assume that the commit that changes the file from one version to another (which is out of the concourse control) is a delimiter between versions - if you do it in the same repo - because the actual version change potentially happened several commits earlier.

With git-tag used as versioning - this issue seems even more severe because intuitively the tag is put on the exact commit.

This is a sample implementation that overcomes this issue with getting as argument the git_resource and taking the sha from it: https://github.com/ggeorgiev/semver-resource/tree/my_ci

setup ci.concourse.ci to run swift tests

The tests on ci.concourse.ci for the resource do not currently run the tests for the swift driver, but do for the git and s3 driver. To be better citizens, we should run all the things!

It would be worth an investigation to have the tests run against a mock swift server.

[Feature request] Git driver: Create directory if it doesn't exist

When I use the git driver and store the version(s) at the root of the repo, the version file is created if it doesn't exist. However, when the "file" property is a path, semver-resource will throw an error:

error setting version: open /tmp/semver-git-repo/environment-a/team-2/application-x: no such file or directory

This means that I need to create the appropriate directory set up manually before using this on a new application. I suppose that s3 driver behavior is different as S3 doesnt care about directories.

Would it be possible to introduce a small "mkdir -p" kind of step before attempting to store the file?

To simplify the creation of "semver" resources, trim leading/trailing whitespace before parsing the semver string

Currently, parsing the content of a semver file fails is there is trailing whitespace (\r, \n, \t, etc.) after the semantic version string. I don't see any specifics about trailing whitespace in the current semver spec, so additional whitespace could be considered "invalid".

It would be much more "user-friendly", however, for the Concourse resource to trim extraneous whitespace from the contents of the semver file. I say this because the most obvious end-user workflow (at least to me) creates an invalid resource.

echo 0.1.0 > myversion`
aws s3 cp myversion s3://mybucket/myversion

... create pipeline referencing the S3 bucket above as a "semver" resource

fly -t my-ci-server set-pipeline -p my-pipeline -c my-pipeline.yml
fly -t my-ci-server check-resource -r my-pipeline/my-semver-resource

The last command will return an error message like:
error checking for new versions: parsing number in bucket: Invalid character(s) found in patch number "0\n"

I was able to work around the error by using echo -n to generate the initial semver file, but that was definitely non-obvious. I also tried creating the file using vi, vim, emacs, nano, and GEdit on my Ubuntu 16.04 system. From what I saw, the default behavior of all of them was to include a trailing newline character in the file.

Determining which level to bump based on outside factors?

Is there a way to determine the bump type based on outside factors within a single job?

For example: I want to bump the minor version when, based on our ticketing system, a new feature is what sparked the pipeline, but bump only the patch version when a bug fix is what triggered things.
I came up with a theoretical solution but it requires multiple jobs and I wanted to make sure I wasn't hacking something that someone already elegantly solved.

tag support for commits in git backend

what would people view be on adding support for tagging the bumping commits by the git driver?

I'm thinking something like an also_tag flag in the resource setup.

Git tag backend

We'd really like to use git tags as the backend for semver so that we can automatically version our code repositories. It's possible to implement this now using a separate semver storage repo and the git resource, however it's possible for the two repositories to fall out of sync.

Opening this here as we've expressed interest in working on it in Slack and were told it would be accepted upstream.

GCS caching means versions are out of date

GCS sets a 3600 second max-age for objects uploaded to GCS buckets:

Unless otherwise specified, the Cache-Control setting for publicly accessible objects is 3600 seconds ... If you allow caching, at download time you might see older versions of objects, even after uploading a newer replacement object, because the older objects remain in the cache for a period of time.

This means that when I put to a semver-resource, I might not be able to successfully get that version back for up to an hour.

This causes some fascinating misbehaviour. The only current workaround is to manually set Cache-Control headers in the bucket. But this means that I have a task following around after every put, which isn't very convenient.

Google suggest setting Cache-Control: private to prevent caching. Can we please see if this improves behaviour?

Put semver silently ignoring error

We have a bump major version job like this:

  - name: garden-runc-bump-major-version
    serial: true
    plan:
    - get: gr-version
      params: {bump: major, pre: rc}
    - put: gr-version
      params: {file: gr-version/number}

It looks like when it was last ran the version 1.2.1-rc.42 was fetched and locally bumped to 1.3.0-rc.1 but we checked the backing s3 file and it was containing 1.2.2-rc.3 which means that most probably the 1.3.0-rc.1 version failed to be pushed back to s3 and the error was silently ignored by semver resource.

Here is a screenshot:
pasted image at 2017_03_13 10_29 am

InvalidAuthentication error intermittently when using correct creds

We are using the semver resource with a GCS bucket. We are seeing intermittent, frequent failures to PUT the semver resource. However, it succeeds sometimes.
We started seeing this two days ago, and 10 out of the last 18 builds have failed for this reason. (and out of the last 11 builds, 8 failed) This is adding significant flakiness to our pipeline.

We get the following error during the put (not during the subsequent get):

error bumping version: InvalidAuthentication: The provided authentication header is invalid.
	status code: 400, request id: , host id: 

Our resource is configured as follows:

- name: release-version
  type: semver
  source:
    key: release-version
    access_key_id: ((gcs-access-key-id))
    secret_access_key: ((gcs-secret-access-key))
    bucket: our-pipeline-store
    region_name: us-east1
    endpoint: storage.googleapis.com

We're doing a put with build params:

- put: release-version
      params: {pre: build}

We are using concourse version 3.8.0.

Missing support for ed25519 SSH keys

While git-resource supports ed25519 keys, semver-resource does not. This was fairly confusing as I was using the same credentials for both resources, but the semver resource was failing as if I had not supplied any key at all:

resource script '/opt/resource/check []' failed: exit status 1

stderr:
Cloning into '/tmp/semver-git-repo'...
Warning: Permanently added 'gist.github.com,192.30.253.119' (RSA) to the list of known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
error checking for new versions: exit status 128

Switching to an RSA key fixes the error.

Please add ed25519 support to this resource so it behaves like git-resource. Thank you!

Swift properties requiring the openstack key is not documented

It is not documented that using semver with the swift driver requires the use of the openstack key.

This does not work.

- name: version
  type: semver
  source:
    driver: swift
    container: bgd-version
    item_name: version
    region: dallas
    ...

Whereas this does:

- name: version
  type: semver
  source:
    driver: swift
    openstack:
      container: bgd-version
      item_name: version
      region: dallas
      ...

"pre" does not like a period in the prefix

Config:

resources:
[...]
  - name: version
    type: semver
    source:
      initial_version: 0.0.0-pr.((pr-name)).0
      driver: git
      branch: pipeline-state
      file: ((pr-name)).version
      uri: https://bitbucket/scm/prj/repo.git
[...]
  - name: increment-pr
    serial_groups: [version]
    plan:
    - aggregate:
      - get: source
        trigger: true
      - get: version
        params: {pre: pr.((pr-name))}
    - put: source
      params:
        repository: source
    - put: version
      params: {file: version/number}

When the increment-pr task runs, I get the same output as the input:
bumped locally from 1.1.2-pr.project-pr-chad-pipeline-test.1 to 1.1.2-pr.project-pr-chad-pipeline-test.1

Any idea why the patch isn't bumping? Does it not like/allow a period?
The value of pre can be anything you like; the value will be pre-pended (hah) to a numeric value.

Version files stored in the same git repository result in commit message confusion

Hello,

Our team stores more than one version in a single repository, where each version is on the same patch version but the build numbers are being bumped frequently. We sometimes see output like this in our git log:

* 593fa78f - bump to 1.12.0-build.124 (10 hours ago) <git>
* 9a3cb01e - bump to 1.12.0-build.12 (11 hours ago) <git>
* 894166d4 - bump to 1.12.0-build.123 (12 hours ago) <git>
* 224c253b - bump to 1.12.0-build.11 (17 hours ago) <git>
* 4d4c4aa2 - bump to 1.12.0-build.122 (17 hours ago) <git>

It would be helpful if these messages were more unique, perhaps by including the path to the version file, to reduce confusion when working in the same repository.

Thanks!
Dave

tag always tags head, it should tag the revision in the `get`

When tagging, it always tags HEAD, instead of the revision pulled with get. This makes it impossible to do something like:

jobs:
- name: first
  steps:
  - get: source
  - task: <build>
- name: second
  steps:
  - get: source
    passed: [first]
  - put: source
    params:
      tag: foo
      only_tag: true

[Question] How to rebuild the latest build version?

Hello.

We use Concourse with semver-resource to build the project.

This is how it is set up:

  • get: git
  • put: version
    params: {bump: patch}
  • put: docker-image

The next question came up.

If a particular build failed, how do we rebuild a project with the same version?

Now I only thought of increasing the version in advance at the end of the build, but it looks clumsy.

Of course, we can remove put steps in the build manifest, but I think there should be a better solution.

Thank you in advance.

git driver is broken when using private_key

In the original code, isPrivateKeyEncrypted is called before Concourse writes the key if it does not yet exist. As a result, isPrivateKeyEncryptedalways returns an err because it is invoking ssh-keygen on a non-existent file, and Concourse bails out. Changing the sequencing restores order to the Force.

https://github.com/VulcanTechnologies/semver-resource/blob/fix-ssh-check/driver/git.go#L203-L221
edited 2020/07/16 to add the patched snippet here which was linked above

func (driver *GitDriver) setUpKey() error {
	_, err := os.Stat(privateKeyPath)
	if err != nil {
		if os.IsNotExist(err) {
			err := ioutil.WriteFile(privateKeyPath, []byte(driver.PrivateKey), 0600)
			if err != nil {
				return err
			}
		} else {
			return err
		}
	}

	if isPrivateKeyEncrypted() {
		return ErrEncryptedKey
	}

	return os.Setenv("GIT_SSH_COMMAND", "ssh -o StrictHostKeyChecking=no -i "+privateKeyPath)
}

This bug was introduced in #77. Like @wjjackson7, I find the lack of unit tests disturbing.

Without those, I'm not going to turn this into a PR, but the workaround is documented for them that stumble here.

Git: automatically create & push semver branch when not present

It would be neat if there were an optional flag to create the specified semver branch if it doesn't exist on remote. I'm imaging something similar to how initial_version: will create & initialize the semver file for you if it doesn't exist.

Has anyone tried something like this / any potential problems with this I might not be seeing?

Thanks!

Git-tag prefix

An optional prefix argument that allows for multiple version tracking in the same git repo.

If you have release branches, or if you track more than one project (or component) you might need to isolate one version tracker from another. This could be easily done with a tag prefix.

Version number as a variable in a shell script

Iโ€™m using a semver resource where the version number is stored in git. Now I want to use the version number in the filename of the build file that is stored in a S3 bucket. We have the pipeline pointing to a separate task-file which points to a shell script. It is in that shell script that I want to use the version as a variable. Now in the shell file the version variable is replaced by the literal argument text [version/number] instead of with the actual version number.

How would I use the version number from the resource in this case?

Missleading error message when incorrect parameters supplied

Hi,

I made the embarassing mistake of missing off the bucket value in the params for the S3 driver. This caused the following error message on out (but not in):

error bumping version: MalformedXML: This happens when the user sends a malformed xml (xml that doesn't conform to the published xsd) for the configuration. The error message, "The XML you provided was not well-formed or did not validate against our published schema."
status code: 400, request_id: 80a9032b:14b587dc17b:b083f:45, host id:

This was against an EMC ECS Appliance - potentially the error message would be different against another S3 compatible blobstore.

If a fundamental setting is omitted for a particular driver it would be great to have a more obvious error message.

Thanks,

Theo

Add bump and pre params to receive from file

I think it's good to decide to what version to go in a plan.
Than I want to add two new params: bump_file and pre_file
With this I can set this params in another task to pass
And only one of this is need to pass: bump or bump_file and pre or pre_file

Thanks

Getting `no version bump specified` when bumping the patch on get

I have configured our build like this:

- get: version
    params: {bump: patch}
  - task: "our_build_task"
       <redacted>
  - put: version

- name: version
  type: semver
  source:
    initial_version: 0.0.0
    driver: git
    uri: [email protected]:<redacted>.git
    branch: master
    private_key: ((github_private_key))
    file: version

Which results in no version bump specified when the put executes. My expectation is that it would bump the version file locally before running the build then put that version. We are running concourse 3.9.2...I just noticed 3.10.0 is available, would an upgrade fix this? and/or should we pull in the semver recourse type rather than use the integrated version?

[Feature Request] allow a flag to 'rediscover' previous versions

We wrote tasks that allow us to quickly roll back a deployment based on a previous version. We also track the currently deployed version so that we can restart / redeploy.

When we 'roll back' a version and push that version to a file using the git driver - the resource will not 'discover' it as new since the version has shown in the past.

version=1.0.0 
// Deploy New
version=2.0.0
// Roll Back
version=1.0.0 

After the last step - version correctly stores 1.0.0 but still reports 2.0.0 in concourse due to this behavior.

it returns the version specified in the file if it is equal to or greater than current version, otherwise it returns no versions.

I would like to propose an option like below

- name: version
  type: semver
  source:
    driver: git
    allow_duplicate_versions: true

If this is a feature that could be included - I can work on a PR

Git driver silently fails to push on retry

We had a build CI in fail to push the bump commit using the git driver even though the put step went green. We think this is due to the retry logic here. In that event that the git commit succeeds but the git push fails, the method is retried but will exit without pushing as it thinks there is nothing to commit.

We also noticed in this that the get step does not do any sanity checking to make sure that the version passed to it via the request matches the version that's located on disk. If this mismatch were caught, we would have seen it fail in the implicit get afterwards. We think the fact that it stopped looking at disk happened in this commit. Adding this behavior might be tricky if you are dealing with storage that's eventually consistent (like S3).

Support EC2 IAM Role SessionToken

I'm using IAM Roles for EC2 security credentials. In trying to use the S3 driver, it looks as though the SessionToken must also be set. Just using the AccessKeyID and SecretAccessKey values results in an error:

resource script '/opt/resource/check []' failed: exit status 1

stderr:
error checking for new versions: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
    status code: 403, request id: XXXXXXXXXXXXXXXX

I didn't see a way to provide the SessionToken to NewStaticCredentials.

No such device or address error in semver and git resources.

I get this error:

resource script '/opt/resource/check []' failed: exit status 1

stderr:
Cloning into '/tmp/semver-git-repo'...
fatal: could not read Username for 'https://bengtssondd.it': No such device or address
error checking for new versions: exit status 128`

On my:

  • MacOS v10.13.6
  • Docker v18.06.1-ce-mac73
  • Docker compose v1.22
  • Concourse CI v4.2.1

The config for semver in my pipeline YAML is:

- name: version
  type: semver
  source:
    branch: version
    driver: git
    file: version
    initial_version: 1.2.1
    password: ((myuser_pass))
    uri: https://bengtssondd.it/git/healops.git
    username: myuser

I have tried:

  • Downgradring to v3.10 of ConourseCI where it used to work. Now it doesn't
  • Thinking about what I've changed:
    -- Using a docker-compose v3.2 file
    -- Postgres v10.5
  • I tried again on another Macbook I have. And no error. It works. But why in the world. It do have a newer version of Git installed. But is that really it? Does the semver and git resources really use the local version of Git? What is going on here? Would be great to get a clearer view on this issue. Thank you.
  • Tried updating Git on the non-working Mac. Even did a full Docker restart on MacOS. And a full re-install of ConcourseCI, removed the Postgres volume.

N.B. 2FA is not enabled on the Git setup I'm using.


What in the world could be wrong?

Release "pre without version" feature

When is the next release planned? I would like to use the "pre without version" feature, but I would have to use the "dev" tag of the semver resource docker in order to use it, since the current latest version is 5 months old and predates that feature.

Git driver works very slow with a very very long history repositories

It takes up to 5 minutes or even more when using semver-resource's git driver to clone a repository with a very very long history. Git driver can be improved with following options:

  1. Using git --single-branch option to clone only version's branch history
  2. Enable using of git --depth option to make a shallow clone of repository

I'm going to propose a PR with those changes

semver-resource doesn't supports "https_tunnel"

While we configuring pipeline with semver-resource, we noticed this resource doesn't support https_tunnel. As per our security requirement SSH over proxy is disabled so we need this https_tunnel to make it work.
Any work around we can do?

stderr:
Cloning into '/tmp/semver-git-repo'...
ssh: connect to host github.com port 22: Operation timed out
fatal: Could not read from remote repository.

Add required AWS permissions for S3 bucket to docs

I am trying to use this resource with an S3 bucket and needed to add the required permissions to my AWS user. I got it to work by using the list of permissions mentioned on https://github.com/concourse/s3-resource:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SemverReadWrite",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::BUCKET_NAME/*",
                "arn:aws:s3:::BUCKET_NAME"
            ]
        }
    ]
}

Perhaps this information should be included in the README?

How to define build ID in meta?

Hi and thank you for this project!
According to SemVer 2.0 we can define build ID:

Build metadata MAY be denoted by appending a plus sign and a series of dot separated identifiers immediately following the patch or pre-release version. Identifiers MUST comprise only ASCII alphanumerics and hyphen [0-9A-Za-z-]. Identifiers MUST NOT be empty. Build metadata SHOULD be ignored when determining version precedence. Thus two versions that differ only in the build metadata, have the same precedence. Examples: 1.0.0-alpha+001, 1.0.0+20130313144700, 1.0.0-beta+exp.sha.5114f85.

How I can do it with semver-resource?

Add a source parameter to avoid dump "pre"

Our users are heavily using semver resource to manage versions.

Recently some users raised requirement to me. They want to generate version like 1.0.0-SNAPSHOT, but semver seems to not support that. If we use pre, generated version will be 1.0.0-SNAPSHOT.1, and additional pre-version is appended.

After digging deeper, we found that the underlying blang/semver also support a Build field, but the problem is that Build is appended with a + sign, like 1.0.0+Build, and the semver standard requires "+" before Build. So we don't want to break the standard.

But the standard doesn't require "pre" to have extra version appended. Thus we are thinking a solution to add a parameter "no-pre-version" to source of semver resource. If this flag is turned on, then just append pre to version without attaching an extra version.

If this solution is ok, we can raise a PR. @vito

If git fails to push, it will not stop retrying

Had a situation, where the user that was pushing to our git repo didn't have push rights to a restricted branch. As a result, the step was running over and over again. There was no helpful information as to what was happening.

I think it would be nice if errors from git were surfaced to the logs and retry had a limit.

Support S3 server side encryption (SSE)

It appears resource-semver, unlike resource-s3, is unable to write to S3 buckets which have a policy requiring encryption. This is a problem for companies that require all S3 assets use encryption (such as one of my clients).

Here is an example policy that might be applied to the S3 bucket:

{
    "Id": "PutObjPolicy",
    "Statement": [
        {
            "Action": "s3:PutObject",
            "Condition": {
                "StringNotEquals": {
                    "s3:x-amz-server-side-encryption": "AES256"
                }
            },
            "Effect": "Deny",
            "Principal": "*",
            "Resource": "arn:aws:s3:::NAME-OF-YOUR-BUCKET/*",
            "Sid": "DenyIncorrectEncryptionHeader"
        },
        {
            "Action": "s3:PutObject",
            "Condition": {
                "Null": {
                    "s3:x-amz-server-side-encryption": "true"
                }
            },
            "Effect": "Deny",
            "Principal": "*",
            "Resource": "arn:aws:s3:::NAME-OF-YOUR-BUCKET/*",
            "Sid": "DenyUnEncryptedObjectUploads"
        }
    ],
    "Version": "2012-10-17"
}

resource-s3 is able to write to a bucket with the above policy, it would be great if resource-semver could too.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.