Can we have file put to support definition of filenames using regex about s3-resource HOT 6 CLOSED

Hi there!

We use Pivotal Tracker to provide visibility into what our team is working on. A story for this issue has been automatically created.

The current status is as follows:

• #126722787 Can we have file put to support definition of filenames using regex

This comment, as well as the labels on the issue, will be automatically updated as the status in Tracker changes.

from s3-resource.

Can you provide an example where a glob isn't good enough? I personally prefer to avoid regexes where possible.

from s3-resource.

Sure. Lets say that you're downloading a pivnet tile, and extracting it and uploading the releases to s3. For example: say you're downloading PCF 1.6.24.

Because of the naming convention of the releases is to include the version number in the file name, this makes it difficult to specify the exact file, also if you have a bunch of files that are:

cf-{version numbers}.tgz
cf-autoscaling-{version numbers}.tgz

If there was just the one unique file name, you could do cf-*.tgz, but this also matches other files.

Also because there may be different numbers of versions number ie:

cf-239.tgz
cf-239.4.tgz

You can't use wildcards to do both of these, you would HAVE to change the pipeline for any format change of the version number.

The final reason is that you can specify a regex to GET the file, why not have one to PUT the file. (especially since you can use the same regex for both, otherwise you have to have a glob and a regex).

resource_types:

• name: storecfrelease
  type: s3
  source:
  bucket: hrconcoursereleases
  regexp: releases/cf-\d+(.\d+)*.tgz

jobs:

• put: storecfrelease
  input:
  • name: releases
    params:
    file: releases/cf-???.??.?.tgz
    acl: public-read

It's quite likely that other software development pipelines may have version numbers in the files they're uploading to s3. Using regex would mean one less thing to cause a pipeline to fail in the use case you change version numbers similar to how we pack the pcf tile..

from s3-resource.

Yeah, that sounds pretty annoying. I'm still not convinced regexes are the best solution though. They'd work, but IMO their use tends to raise more questions than they answer.

When another engineer is ramping up and sees the regexp cf-\d+.tgz it won't be clear whether it was intentional for that to not match cf-239.4.tgz or if it was an oversight. It also won't be clear if was to avoid matching cf-autoscaling-123.tgz, cf-metadata.tgz, etc.

The use of regexes in source in contrast feels a lot more clear. We need capture groups to know which bit is the version segment, and there's only one piece of state the regexp will be running against (the contents of the bucket). Compared to put, whose data is dynamic, and harder to predict as an engineer rolling on.

An alternative would be to switch both to a globbing/templating syntax like cf-{version}.tgz or cf-with-sha.*.{version}.tgz. That wouldn't support the case of having cf-239.tgz and cf-239.4.tgz though - is that a real thing?

from s3-resource.

I don't think i explained my intentions very well. Let me try a different way.

Lets say you have a pipeline to upgrade an existing pcf deployment from a pivnet tile. (ie on Azure or an IaaS without ops manager).

The current pipeline has cf-???.tgz (for cf-350.tgz), however when you bump the pcf tile version, it contains release cf-380.4 which means you'd have to change the pipeline, similarly if the next version you deploy is say cf-390.40, or cf-390. If you could create a regex then you could iterate over the expression and ensure it deploys for all possibilities.

My goal would be to split and upload versions without having to either:
a. have a script to separate out the releases.
b. modify the pipeline.

That being said if you had a tempting syntax, that would be good where version would be any combination of 0-9 and (-, .) . However there could be an where someone tries to label a file "ALPHA, BETA" etc.

from s3-resource.

We are considering this issue dead due to inactivity. Please feel free to reopen a new issue if this is still a problem. Thanks!

from s3-resource.