Steps to reproduce

1. Install Go 1.22.2
2. Run task test

Expected result

Tests pass, 100% coverage

Actual behavior

Tests pass, 73.5% coverage

Miscellany

task deps should probably look to installing @latest of all dependent packages, rather than locking in at what appears to be arbitrary versions.

Python users who get Gilt from PyPI may still want to use the new version; we should give them an "easy" upgrade path via pip and the like.

There is probably some wisdom to be gleaned from pip-binary-factory on how to assemble a Python package containing non-Python files.

Story time!

We have devised a workflow for vendoring in changes that goes something like:

• Pull in files from upstream
• Run git apply over the top of them to localize for our needs
• Fail if the patch does not apply cleanly

When absorbing new versions from upstream, we do a three-way merge of "old upstream", "new upstream", and our localizations. To do this with Gilt, it is necessary to temporarily comment out the post-commands from our Giltfile.yaml to get a "pristine" copy.

Gilt could help with this by providing a command-line switch that only runs the file overlay steps and avoids running post-commands.

TL;DR: we'd like a --no-commands switch, or something similar.

Spotted when trying to vendor in a fresh change; reproduction is a bit weird:

Steps to reproduce

Setup

mkdir repo cache target
cd repo
git init .
touch monkey
git add monkey
git commit -m 'i has a repo'
git tag -am v1 v1
cd ..
cat << EOF > target/Giltfile.yaml
giltDir: $(pwd)/cache
debug: true
repositories:
  - git: file://$(pwd)/repo
    version: v1
    sources:
    - src: monkey
      dstFile: monkey
EOF

Verify setup

cd target
gilt overlay

Finish setup

cd ../repo
touch banana
git add banana
git commit -m 'my hands are typing words'
git tag -am v2 v2
cd ..
cat << EOF > target/Giltfile.yaml
giltDir: $(pwd)/cache
debug: true
repositories:
  - git: file://$(pwd)/repo
    version: v2
    sources:
    - src: monkey
      dstFile: monkey
    - src: banana
      dstFile: banana
EOF

Test

cd target
gilt overlay

Expected result

target directory contains monkey and banana

Actual behavior

DBG result output="fatal: invalid reference: v2\n"
[...]
ERR error overlaying repositories err="exit status 128"

Workaround

cd ../cache/cache/*
git fetch --tags

...will update the cache with current data, and the overlay will succeed.

I'd like to be able to specify a tag name instead of a commit id for the version parameter.

I see this can be done trivially by relaxing the regex in schema.go, but I think I can possibly do a bit better than that, maybe by making version and tag into a oneOf.

Is this a change you'd be interested in taking - I can't tell which of the features of python-gilt didn't come across "by design".

My gut feeling is that tags are "probably OK" and branches are "probably not OK" in terms of reproducibility.

Gilt fails to run when installed on:

• Ubuntu < 23.10
• Debian < 12
• Alpine < 3.17
• RedHat < 8.7

It yields the error message:

ERR error overlaying repositories err="exit status 128"

This is not a Gilt error per se, but an incompatibility with the antediluvian versions of Git that come installed on these old, LTS Linux distros

Steps to reproduce

docker run --rm -it ubuntu:jammy
apt update
apt -y install git
git clone --bare --filter=blob:none --origin gilt https://github.com/docker-library/busybox.git /home/jenkins/.gilt/clone/cache/https---github.com-docker-library-busybox.git

Expected result

Cloning into bare repository '/home/jenkins/.gilt/clone/cache/https---github.com-docker-library-busybox.git'...
remote: Enumerating objects: 2132, done.
remote: Counting objects: 100% (1058/1058), done.
remote: Compressing objects: 100% (487/487), done.
remote: Total 2132 (delta 458), reused 900 (delta 442), pack-reused 1074
Receiving objects: 100% (2132/2132), 286.35 KiB | 1.86 MiB/s, done.
Resolving deltas: 100% (618/618), done.

Actual behavior

fatal: --bare and --origin gilt options are incompatible.

Workaround

• Install/use Git 2.39 or later. This is the default in Debian 12, Ubuntu 23.10, Alpine 3.17, etc.
• Use the equivalent/compatible command:

git -c clone.defaultRemoteName=gilt clone --bare --filter=blob:none https://github.com/docker-library/busybox.git /home/jenkins/.gilt/clone/cache/https---github.com-docker-library-busybox.git

Miscellany

Ubuntu 20.04 is supported until 2025, and 22.04 is supported until 2027, so this isn't the kind of thing that can be easily laughed off.

Apple has shipped comparatively-modern versions of Git for a long while now, making this issue difficult to spot in a development environment.

This issue very likely disappears completely of go-git ever ships a version that makes Issue #72 plausible.

When copying files, Gilt's workflow is:

• Opens the source file for reading
• Opens the destination file for writing according to the current process umask
• Copies the file contents
• Gets permissions bits from the source file
• Applies the permissions to the destination file

This is wide-open for a TOCTOU attack.

This is fairly low-priority, since there's little in the way of custom handling being done, but at least a cursory attempt should be made to ensure that reading config files works as intended.

Steps to reproduce

1. Run go-gilt and redirect its output, run inside a Jenkins job, etc.
2. Look at the logs

Expected result

...
bin/go-gilt --gilt-file apps/athens/Giltfile.yaml overlay 
11:58PM INF acquiring lock lockfile=/home/jenkins/.gilt/clone/gilt.lock
...

Actual behavior

...
bin/go-gilt --gilt-file apps/athens/Giltfile.yaml overlay 
�[2m11:58PM�[0m �[92mINF�[0m acquiring lock �[2mlockfile=�[0m/home/jenkins/.gilt/clone/gilt.lock
...

Steps to reproduce

go install github.com/retr0h/gilt@latest

Expected result

It installs v2.1.1

Actual behavior

go: github.com/retr0h/gilt@latest: version constraints conflict:
	github.com/retr0h/[email protected]: parsing go.mod:
	module declares its path as: github.com/retr0h/go-gilt
	        but was required as: github.com/retr0h/gilt

LOLWUT

$ curl -sL https://proxy.golang.org/github.com/retr0h/gilt/@latest
{"Version":"v1.0.2","Time":"2023-12-27T21:44:45Z","Origin":{"VCS":"git","URL":"https://github.com/retr0h/gilt","Ref":"refs/tags/v1.0.2","Hash":"c9f7195faa37db6e355890eca82963d34b6fdb55"}}

Something has gone pretty wrong.

Steps to reproduce

Go to https://github.com/retr0h/gilt

Expected result

All links are up to date

Actual behavior

Miscellany

LOL

There's an indirect requirement for the git CLI to be installed and in the $PATH for Gilt to work at all. It's also an unspoken requirement for the git to be 2.20 or newer (that isn't a high bar to clear, newer versions than that are even in Debian LTS)

go-git supports bare clones, and has enough worktree support for what Gilt is trying to get done, so it could be folded in and Gilt could handle everything on its own.

This is unfortunately blocked on a lack of filter support for clones in go-git: go-git/go-git#713

Spotted while trying to release v2.2.1:

https://github.com/retr0h/gilt/actions/runs/8964038315/job/24615206856

Building wheels fails:

Run task build:wheel
task: [build:wheel] python/dist2wheel.py
Traceback (most recent call last):
  File "/home/runner/work/gilt/gilt/python/dist2wheel.py", line 183, in <module>
    Wheels().create_all()
  File "/home/runner/work/gilt/gilt/python/dist2wheel.py", line [6](https://github.com/retr0h/gilt/actions/runs/8964038315/job/24615206856#step:8:7)9, in create_all
    if "BrewConfig" in artifact["extra"]:
KeyError: 'extra'
task: Failed to run task "build:wheel": exit status 1

Steps to reproduce

Use this Giltfile.yaml:

---
repositories:
  - git: https://github.com/retr0h/ansible-etcd.git
    version: 77a95b7
    dstDir: .

...and attempt gilt overlay

Expected result

10:03AM INF acquiring lock lockfile=/tmp/.gilt/gilt.lock
10:03AM INF cloning repository=https://github.com/retr0h/ansible-etcd.git dstDir=/tmp/.gilt/cache/https---github.com-retr0h-ansible-etcd.git
10:03AM INF extracting from=/tmp/.gilt/cache/https---github.com-retr0h-ansible-etcd.git version=workflow-cps-2.84 to=/tmp

Actual behavior

10:03AM INF acquiring lock lockfile=/tmp/.gilt/gilt.lock
10:03AM INF cloning repository=https://github.com/retr0h/ansible-etcd.git dstDir=/tmp/.gilt/cache/https---github.com-retr0h-ansible-etcd.git
10:03AM ERR error overlaying repositories err="RemoveAll .: invalid argument"

Workaround

Make use of a temporary directory, and post-commands: Giltfile.yaml

repositories:
  - git: https://github.com/retr0h/ansible-etcd.git
    version: 77a95b7
    dstDir: gilt-tmp
    commands:
    - cmd: bash
      args:
      - '-c'
      - '(cd gilt-tmp ; tar cf - .) | tar xf -'
    - cmd: rm
      args:
      - '-rvf'
      - gilt-tmp

Miscellany

Gilt 1.x allowed this.

Steps to reproduce

Change debug: false to debug: true in the example Giltfile.yaml, and:

task build
export RUN_DIR=dist/go-gilt_$(uname -s | tr '[:upper:]' '[:lower:]')_$(uname -m | sed -e 's/x86_/amd/' -e '/amd/s/$/_v1/')
$RUN_DIR/go-gilt overlay

Expected result

Debug output, since debug: true is set

Actual behavior

No debug output

Workaround

$RUN_DIR/go-gilt --debug overlay

Some idiot replaced the entire caching layer of this thing without first checking if the new one would even work with his existing inputs.

Steps to reproduce

• Attempt to overlay this Giltfile.yaml:

---
repositories:
  - git: [email protected]:bitnami/charts.git
    version: a930cf8b438809fc203f5537898dfa326955dedb
    sources:
    - src: bitnami/fluentd
      dstDir: /tmp/elasticsearch-fluentd/charts/vendor/fluentd
    - src: bitnami/common
      dstDir: /tmp/elasticsearch-fluentd/charts/vendor/common

  - git: [email protected]:elastic/helm-charts.git
    version: c77d1a7ae38c17a553399d397a9fa9f064991803
    sources:
    - src: elasticsearch
      dstDir: /tmp/elasticsearch-fluentd/charts/vendor/elasticsearch
    - src: logstash
      dstDir: /tmp/elasticsearch-fluentd/charts/vendor/logstash

  - git: [email protected]:prometheus-community/helm-charts.git
    version: d5d699cee8b95aa294ea6e691c2a3f9de00b6d79
    sources:
    - src: charts/prometheus-elasticsearch-exporter
      dstDir: /tmp/elasticsearch-fluentd/charts/vendor/prometheus-elasticsearch-exporter

  - git: [email protected]:lebenitza/charts.git
    version: c6597fe0a3ddb11465b38ebe25c19366ef3dfeb6
    sources:
    - src: charts/elasticsearch-curator
      dstDir: /tmp/elasticsearch-fluentd/charts/vendor/elasticsearch-curator

Expected result

The command completes successfully

Actual behavior

11:34AM INF acquiring lock lockfile=/Users/nisimond/.gilt/clone/gilt.lock
11:34AM INF cloning [email protected]:bitnami/charts.git dstDir=/Users/nisimond/.gilt/clone/cache
...
11:34AM INF clone already exists dstDir=/Users/nisimond/.gilt/clone/cache/helm-charts.git
11:34AM INF extracting from=/Users/nisimond/.gilt/clone/cache/helm-charts.git version=d5d699cee8b95aa294ea6e691c2a3f9de00b6d79 to=/Users/nisimond/.gilt/clone/tmp679354367/helm-charts.git
11:34AM ERR error overlaying repositories err="exit status 128"

Miscellany

The test case provides two different flavors of the same problem; in one, there is an old pre-Bitnami fork of the charts.git repo that collides; in the other, there are two entirely separate projects that both have repositories named helm-charts.git.

There is also the case of maintaining a private fork of an active repo, and wishing to maintain caches of both of them concurrently. All of these use cases should be accounted for when resolving this.

Commit d306437 made the handling of these two fields 100% equivalent, so they should be deprecated and combined into a repositories.version field, or similar.

Spotted when trying to vendor in an update:

6:24PM INF clone already exists dstDir=/Users/nisimond/.gilt/clone/cache/[email protected]
6:24PM DBG exec command="git fetch --tags" cwd=/Users/nisimond/.gilt/clone/cache/[email protected]
6:24PM DBG result output="Authenticated to github.com ([192.30.255.113]:22) using \"publickey\".\r\nTransferred: sent 16864, received 662100 bytes, in 1.8 seconds\r\nBytes per second: sent 9142.3, received 358936.7\r\nFrom github.com:kubernetes/ingress-nginx\n * branch              HEAD              -> FETCH_HEAD\n * [new tag]           4e97379b4         -> 4e97379b4\n ! [rejected]          controller-v1.6.4 -> controller-v1.6.4  (would clobber existing tag)\n ! [rejected]          controller-v1.7.1 -> controller-v1.7.1  (would clobber existing tag)\n ! [rejected]          controller-v1.8.2 -> controller-v1.8.2  (would clobber existing tag)\n ! [rejected]          controller-v1.8.4 -> controller-v1.8.4  (would clobber existing tag)\n * [new tag]           controller-v1.8.5 -> controller-v1.8.5\n ! [rejected]          controller-v1.9.0 -> controller-v1.9.0  (would clobber existing tag)\n ! [rejected]          controller-v1.9.3 -> controller-v1.9.3  (would clobber existing tag)\n * [new tag]           controller-v1.9.6 -> controller-v1.9.6\n * [new tag]           helm-chart-4.7.5  -> helm-chart-4.7.5\n * [new tag]           helm-chart-4.9.1  -> helm-chart-4.9.1\n"
6:24PM ERR error overlaying repositories err="exit status 1"

TL;DR: Gilt needs to run git fetch --tags --force when updating cache.

Not sure what happened, as releases have worked perfectly fine previously w/o change. Correct the failing release job:

Error: The process '/opt/hostedtoolcache/goreleaser-action/1.24.0/x64/goreleaser' failed with exit code 1

The docs for Configuration and Usage have fallen a bit behind; get them up to date with all available config options/switches/etc. to reflect the various and sundry ways people can use this thing.

With the new version (2.2.0), the cache does not update itself when running gilt overlay on a already overlayed folder. When the target git repo is updated, I need to clear the cache directory (~/.gilt) first in order to have the latest repo. I believe the code only doing a git fetch instead of git fetch and pull.

Gilt will create a separate, full clone (with working tree!) of the given repository for EVERY version specified in EVERY Giltfile.yaml.

For a large monorepo repository like https://github.com/bitnami/charts, where it is common to track multiple revisions based on which subtree is being used, this means downloading a fresh 3.2GB copy of that repository every time a new revision is mentioned. This almost completely destroys all utility provided by a cache.

There are better ways.

Steps to reproduce

for x in {1..8}
do
    go run main.go overlay &
done
wait

Expected result

All commands try to grab the lock, patiently wait their turn, run, and exit successfully.

Actual behavior

7 of the commands will say:

ERR error overlaying repositories err="could not acquire lock on gilt.lock: fslock: lock is held"

...and exit 1, and the eighth will grab the lock and run to completion

I installed gilt:

go install github.com/retr0h/gilt/v2@latest
# go: downloading github.com/retr0h/gilt v1.0.2
# go: downloading github.com/retr0h/gilt/v2 v2.2.0

Version output

go/bin/gilt version
# {"version":"dev","commit":"none","date":"unknown"}

Expected something like

go/bin/gilt version
# {"version":"2.2.0","commit":"185799bf","date":"2024-03-06"}

Steps to reproduce

go install github.com/retr0h/gilt/v2@latest

Expected result

go: downloading github.com/retr0h/gilt/v2 v2.2

Actual behavior

go: downloading github.com/retr0h/gilt v1.0.2
go: downloading github.com/retr0h/gilt/v2 v2.1.2

Miscellany

pip install python-gilt DOES install version 2.2, so the publishing job does seem to have run.

Tools like ruff have shown that it is possible to have non-Python tools that leverage the Python ecosystem (installation via pip, automatic hash checking, etc.), and can even become wildly popular in doing so.

Gilt should attempt to take a page from that playbook, and as a side-effect, unify the schism between implementations.

I'd advocate for putting everything in the "other" repo and retiring this one, but Your House, Your Rules™.

Spotted with Gilt 2.2.0:

Steps to reproduce

Setup

Use this Giltfile.yaml:

---
repositories:
  - git: https://github.com/retr0h/gilt.git
    version: main
    sources:
      - src: README.md
        dstFile: /tmp/chet/says/this/does/not/work/README.md

Test

gilt overlay

Expected result

Command runs successfully, copy of the README.md is in a deep dir under /tmp

Actual behavior

Hilarity ensues:

10:10AM ERR error overlaying repositories err="open /tmp/chet/says/this/does/not/work/README.md: no such file or directory"

Workaround

Use sources.dstDir instead, it works as expected.

Steps to reproduce

Use this Giltfile.yaml:

---
giltDir: ~/.gilt/clone
repositories:
  - git: https://github.com/lorin/openstack-ansible-modules.git
    version: 2677cc3
    sources:
      - src: neutron_router
        dstFile: /tmp/library/neutron_router.py

Run gilt overlay

Expected result

It creates a /tmp/library/neutron_router.py file

Actual behavior

2:45PM ERR error overlaying repositories err="open /tmp/library/neutron_router.py: permission denied"

Miscellany

It is attempting to create the missing target directory with the same permissions as the destination file. the EXACT SAME permissions as the destination file:

$ ls -ld /tmp/library
drw-r--r--  2 nisimond  wheel  64 Jun 19 14:45 /tmp/library/