lunarway / shuttle Goto Github PK
View Code? Open in Web Editor NEWCLI for handling shared build and deploy tools between projects no matter what technologies the projects are using
License: Apache License 2.0
CLI for handling shared build and deploy tools between projects no matter what technologies the projects are using
License: Apache License 2.0
It would be nice if one could open the plan repository from shuttle it self. Something like below.
It's kind of related to #7 in that it will provide access to the documentation.
# open repository webview, eg. GitHub, for current plan
shuttle plan --web
To aid troubleshooting issues it would be great to be able to get shuttle version, current plan and version etc. output.
Something like go env
in the go tool chain or npm version
for Node.js.
$ go env
GOARCH="amd64"
GOBIN="/Users/bjornsorensen/go/bin"
GOCACHE="/Users/bjornsorensen/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/bjornsorensen/go"
...
{ npm: '6.4.1',
ares: '1.14.0',
cldr: '33.1',
http_parser: '2.8.0',
icu: '62.1',
modules: '64',
napi: '3',
nghttp2: '1.33.0',
node: '10.11.0',
openssl: '1.1.0i',
tz: '2018e',
unicode: '11.0',
uv: '1.23.0',
v8: '6.8.275.32-node.28',
zlib: '1.2.11' }
When looking into an issue we can derive much information of such reports. I'm thinking something like
$ shuttle config
version 0.5.2
Environment:
LW_PATH=/user/...
Plan:
git://[email protected]:LunarWay/lw-shuttle-go-plan.git c331cbb3b71970dc6a3afabcd9320fb070bb719d
Maybe even let plans specify specific information related to the plan, ie. go env
for Go plans etc.
If one runs a script with arguments that are not specified for the script it is silently ignored.
This makes it error prone for non-required fields.
scripts:
test:
args:
- name: version
actions:
- shell: echo "Version $version"
$ shuttle run test versoin=123
Version
$ echo $?
0
I would expect the above command to at least exit with a non-zero status.
Preferable with a message like unknown argument "versoin"
The idea is to use --arg=value
instead of arg=value
.
This is to rely on cobra instead of our own validation logic. This will improve usability and be a more homogenous design
We need to support using a specific version of a plan.
It could be a specific git commit, git tag or similar.
Any scripts that output a percentage sign is wrongly outputtet as it is recognised as a placeholder.
With the following shuttle.yaml
plan: false
scripts:
percentage:
actions:
- shell: echo '%'
Running shuttle run percentage
outputs
$ shuttle run percentage
shell: echo '%'
%!(NOVERB)
I expected it to output the following instead.
$ shuttle run percentage
shell: echo '%'
%
We should setup travis to run the tests and do automated releases using goreleaser https://goreleaser.com/ci/
Auto completion for dynamic scripts and arguments would enhance the experience using shuttle
when trying to fire
curl -LO https://github.com/lunarway/shuttle/releases/download/$(curl -Lso /dev/null -w %{url_effective} https://github.com/lunarway/shuttle/releases/latest | grep -o '[^/]*$')/shuttle-darwin-amd64
throws the following error:
parse error near
)'`
This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.
These updates have all been created already. Click a checkbox below to force a retry/rebase of any.
dagger.io/dagger
, github.com/go-cmd/cmd
, github.com/spf13/cobra
, go
, golang.org/x/exp
, golang.org/x/mod
, golang.org/x/sync
).github/workflows/build.yml
actions/checkout v4
actions/setup-go v5
actions/cache v4
.github/workflows/release-drafter.yml
release-drafter/release-drafter v6
.github/workflows/release.yml
actions/checkout v4
actions/setup-go v5
actions/cache v4
goreleaser/goreleaser-action v6
go.mod
go 1.21
go 1.22.4
github.com/Masterminds/sprig/v3 v3.2.3
github.com/cli/safeexec v1.0.1
github.com/go-cmd/cmd v1.4.2
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510@e7afc7fbc510
github.com/pkg/errors v0.9.1
github.com/spf13/cobra v1.8.0
github.com/stretchr/testify v1.9.0
golang.org/x/exp v0.0.0-20240604190554-fc45aab8b7f8@fc45aab8b7f8
dagger.io/dagger v0.11.6
github.com/AlecAivazis/survey/v2 v2.3.7
github.com/google/uuid v1.6.0
github.com/iancoleman/strcase v0.3.0
github.com/matishsiao/goInfo v0.0.0-20210923090445-da2e3fa8d45f@da2e3fa8d45f
github.com/otiai10/copy v1.14.0
golang.org/x/mod v0.18.0
golang.org/x/sync v0.7.0
gopkg.in/yaml.v2 v2.4.0
pkg/executors/golang/codegen/testdata/patch/replace_existing/.shuttle/actions/tmp/go.mod
go 1.21.4
pkg/executors/golang/codegen/testdata/patch/replace_existing/go.mod
go 1.21.4
pkg/executors/golang/codegen/testdata/patch/root_module/.shuttle/actions/tmp/go.mod
go 1.21.4
pkg/executors/golang/codegen/testdata/patch/root_module/go.mod
go 1.21.4
pkg/executors/golang/codegen/testdata/patch/root_workspace/.shuttle/actions/tmp/go.mod
go 1.21.4
pkg/executors/golang/codegen/testdata/patch/root_workspace/go.mod
go 1.21.4
pkg/executors/golang/codegen/testdata/patch/root_workspace/other/subpackage/go.mod
go 1.21.4
pkg/executors/golang/codegen/testdata/patch/root_workspace/subpackage/go.mod
go 1.21.4
pkg/executors/golang/compile/testdata/simple/actions/go.mod
go 1.18
pkg/executors/golang/discover/testdata/child/actions/go.mod
go 1.18
pkg/executors/golang/discover/testdata/parent/actions/go.mod
go 1.18
pkg/executors/golang/executer/testdata/child/actions/go.mod
go 1.18
pkg/executors/golang/executer/testdata/parent/actions/go.mod
go 1.18
.github/workflows/build.yml
golang 1.23.0
.github/workflows/release.yml
golang 1.23.0
pkg/executors/golang/compile/compile.go
golang 1.23.0-alpine
This effectively means that you cannot use shuttle v0.20.0 outside of a shuttle project, as well as shuttle get
being polluted with
Cloning ... <variable>
during the first run of shuttle on a project.
When shuttle scripts call other shuttle scripts, it gets hard to understand what went wrong with the current error logging.
See this example from the root of this repo.
$ shuttle --project examples/moon-base run say-hi && shuttle --project examples/moon-base run crash
shell: echo "test"
test
shell: exit 1
shuttle failed
Failed executing shell script `exit 1`
Exit code: 1
When you get an error from git, using a git-plan, all information about the error is hidden. You only get the status code from git. Would be nice with the full git error printed out
Hello! I'm trying out shuttle, but experiencing some weird behavior when using a remote git plan:
My shuttle.yaml
looks like this:
plan: [email protected]:<my-org>/<my-repo>
When I run shuttle plan
I get the following output:
Error: exit code 2 - Failed to open plan configuration: open /home/zander/<my-org>/<my-repo>/[email protected]:<my-org>/<my-repo>/plan.yaml: no such file or directory
Make sure you are in a project using shuttle and that a 'shuttle.yaml' file is available.
Usage:
... (help-output) ...
shuttle failed
Failed to open plan configuration: open /home/zander/<my-org>/<my-repo>[email protected]:<my-org>/<my-repo>/plan.yaml: no such file or directory
Make sure you are in a project using shuttle and that a 'shuttle.yaml' file is available.
Which I would expect to work as per the readme.
If I change the plan to the git://
syntax it works as expected:
# shuttle.yaml
plan: git://[email protected]:<my-org>/<my-repo>
# terminal
❯ shuttle plan
git://[email protected]:<my-org>/<my-repo>%
Am I doing something wrong? Or is the documentation not correct?
PS. Shuttle looks cool, we are thinking about adopting it for our build needs, thanks for doing open-source! 😄
When generating templates using the following template functions, the order of the values are not deterministic.
This becomes clear when the results of tempalted files are committed where the different order will indicates changes of (properly) non-functional changes.
It will prompt for values with default values
It would be great if there was a way to list all available templates for a given plan.
This could help users identify unexpected overwrites of plan templates and also help with discovering what templates are available shuttle template
.
Maybe something like below. The most important thing is the name of the template and if it's overwritten by another template.
$ shuttle template --list
Plan templates:
- deployment.tmpl
- Dockerfile.tmpl (overwriting plan)
- secret.tmpl
If run with the --verbose
(-v
) flag we could show the paths to each template. This could help in understanding where to look when updating or trying to understand them.
$ shuttle template --list -v
Plan templates:
- deployment.tmpl $plan/templates/deployment.tmpl
- Dockerfile.tmpl $project/templates/Dockerfile.tmpl (overwriting $plan/templates/Dockerfile.tmpl)
- secret.tmpl $project/templates/secret.tmpl
I tried to run shuttle get service
(version v0.23.0) but got an error that the command was not known to shuttle. When i ran shuttle --help
it wasn't showing up. I looked at the ./cmd/get_test.go file and there it test for this command. I checked the pipeline and the test succeeds.
I then created a samle shuttle.yaml
file and ran the command again, and then it worked. When i run shuttle --help
shuttle suddenly displays the get
command.
Here's the diff on the --help output depending on shuttle finds a shuttle.yaml
file or not. Based on the output below, get
is not the only feature that goes missing depending whether there's a shuttle file or not.
Is this by design, and error, is shuttle bending the fabric of spacetime or is this some kind of metamorphic feature?
If would be great if arguments for scripts were validated before issuing the command.
It would require some syntax for writing requirements of an argument in a plan. This could very well be the same syntax required for #6 for validating the plan schema.
This could add more descriptive help messages as well (#14) as we define what values are allowed up front.
A naive implementation would be something like below with key validation
on the argument.
plan: git://[email protected]:LunarWay/lw-shuttle-go-plan.git
scripts:
my_script:
description: a description
args:
- name: foo
required: true
validation:
- someValue
- someOtherValue
actions:
- shell: './run.sh'
$ shuttle run my_script foo=someWrongValue
Argument foo contains an invalid value "someWrongValue". Allowed values: "someValue", "someOtherValue"
Running shuttle version
crashes with error:
panic: unable to redefine 'c' shorthand in "version" flagset: it's already used for "commit" flag
goroutine 1 [running]:
github.com/spf13/pflag.(*FlagSet).AddFlag(0xc4200a4500, 0xc4200b0fa0)
/home/travis/gopath/src/github.com/spf13/pflag/flag.go:862 +0x72b
github.com/spf13/pflag.(*FlagSet).AddFlagSet.func1(0xc4200b0fa0)
/home/travis/gopath/src/github.com/spf13/pflag/flag.go:875 +0x71
github.com/spf13/pflag.(*FlagSet).VisitAll(0xc4200a4800, 0xc4200f1c10)
/home/travis/gopath/src/github.com/spf13/pflag/flag.go:278 +0xa0
github.com/spf13/pflag.(*FlagSet).AddFlagSet(0xc4200a4500, 0xc4200a4800)
/home/travis/gopath/src/github.com/spf13/pflag/flag.go:873 +0x50
github.com/spf13/cobra.(*Command).mergePersistentFlags(0x148c6e0)
/home/travis/gopath/src/github.com/spf13/cobra/command.go:1495 +0x90
github.com/spf13/cobra.(*Command).InitDefaultHelpFlag(0x148c6e0)
/home/travis/gopath/src/github.com/spf13/cobra/command.go:906 +0x2f
github.com/spf13/cobra.(*Command).execute(0x148c6e0, 0x14b0878, 0x0, 0x0, 0x148c6e0, 0x14b0878)
/home/travis/gopath/src/github.com/spf13/cobra/command.go:686 +0x59
github.com/spf13/cobra.(*Command).ExecuteC(0x148c940, 0x13077ec, 0x26, 0x148c6e0)
/home/travis/gopath/src/github.com/spf13/cobra/command.go:852 +0x30a
github.com/spf13/cobra.(*Command).Execute(0x148c940, 0x1003a8c, 0xc420018118)
/home/travis/gopath/src/github.com/spf13/cobra/command.go:800 +0x2b
github.com/lunarway/shuttle/cmd.Execute()
/home/travis/gopath/src/github.com/lunarway/shuttle/cmd/root.go:44 +0x31
main.main()
/home/travis/gopath/src/github.com/lunarway/shuttle/main.go:8 +0x20```
Shuttle golang actions fail because it cannot be utilized inside a go workspace.
A plan could be enhanced with some good visible documentation. One way of doing this is simply making the CLI able to print the README.md
of the plan which would make the readme automatically work with both git repos and CLI.
Consider using https://github.com/ec1oud/blackfriday
Hello!
I'm really enjoying using shuttle, so much in fact that I've been trying to add some interactivity to some of my scripts.
Though I have found that it is currently not possible to have interactive sessions through shuttle.
For example if I call a script that uses gum choose foo bar
the command simply hangs forever.
Another example: Running an interactive docker container docker run --rm -it alpine
returns the error the input device is not a TTY
.
I think this could be a valuable addition to shuttle that would open up for using the tool in new and interesting ways.
Thank you for all of your work and for open sourcing shuttle :-)
Currently if shuttle is called outside of a project with shuttle.yaml, it will complain that it can't find a shuttle.yaml file. Which is reasonable in 99% of cases, the 1% is where we want to upload shuttle telemetry data (if wanted)
It is hard and error-prone to use the vars
system. What vars
can be used? Did I misspell some of them?
Add some level of schema for the vars
, so required vars and optional vars are easily validated
Line 1 in 3277246
we are currently missing license holder for script source copy right claim.
When using shuttle to start a long-running process, eg. web-server, signals from the terminal are not forwarded to the process. This leaves sub-processes running invisible and detached from the client's terminal.
Define a shuttle.yaml
that starts a docker image
plan: false
scripts:
test:
actions:
- shell: docker run --rm -i nginx
Run the shuttle script and stop after nginx has started with ctrl + c
.
$ shuttle run test
shell: docker run --rm -i nginx
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
^C%
Now verify that the docker image is in fact still running
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
eddda9d31c2a nginx "/docker-entrypoint.…" 4 seconds ago Up 3 seconds 80/tcp ecstatic_moser
Expected behaviour is that the docker image was terminated.
It's ignored silently when putting in unsupported properties in the plan.yaml
and shuttle.yaml
. Would be great if it could be handled with a validation "warning" (maybe not crashing)
When running any shuttle command in a directory without a shuttle.yaml
file it panics.
$ mkdir t
$ cd t
$ shuttle ls
panic: yamlFile.Get err #open /Users/bjornsorensen/repo/t/shuttle.yaml: no such file or directory
goroutine 1 [running]:
github.com/lunarway/shuttle/pkg/config.(*ShuttleConfig).getConf(0xc0000ce270, 0xc000012640, 0x48, 0x0)
/home/travis/gopath/src/github.com/lunarway/shuttle/pkg/config/shuttleconfig.go:70 +0x2ad
github.com/lunarway/shuttle/pkg/config.(*ShuttleProjectContext).Setup(0xc0000ce240, 0xc000012640, 0x48, 0x13238dc, 0x4, 0x13238dc, 0x4, 0x0, 0x0, 0x0, ...)
/home/travis/gopath/src/github.com/lunarway/shuttle/pkg/config/shuttleconfig.go:38 +0x63
github.com/lunarway/shuttle/cmd.getProjectContext(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/home/travis/gopath/src/github.com/lunarway/shuttle/cmd/root.go:71 +0x1d7
github.com/lunarway/shuttle/cmd.glob..func6(0x15659e0, 0x158a0b8, 0x0, 0x0)
/home/travis/gopath/src/github.com/lunarway/shuttle/cmd/ls.go:26 +0x37
github.com/spf13/cobra.(*Command).execute(0x15659e0, 0x158a0b8, 0x0, 0x0, 0x15659e0, 0x158a0b8)
/home/travis/gopath/src/github.com/spf13/cobra/command.go:766 +0x2cc
github.com/spf13/cobra.(*Command).ExecuteC(0x15665c0, 0x8, 0x9, 0xc0000bc380)
/home/travis/gopath/src/github.com/spf13/cobra/command.go:852 +0x2fd
github.com/spf13/cobra.(*Command).Execute(0x15665c0, 0x1566360, 0xc0000dbf20)
/home/travis/gopath/src/github.com/spf13/cobra/command.go:800 +0x2b
github.com/lunarway/shuttle/cmd.Execute()
/home/travis/gopath/src/github.com/lunarway/shuttle/cmd/root.go:49 +0x3f
main.main()
/home/travis/gopath/src/github.com/lunarway/shuttle/main.go:8 +0x20
We should handle the error more gracefully. An error log with the message No shuttle.yaml present in directory
should suffice.
Magefile is a make replacement for go, it works by having go files act as commands.
such that:
cmd/run_task.go
mage run_task
will work. It does this automatically without having to register the command anywhere, which greatly helps discovering and building commands. This combined with plans could be great boon for quickly adding actions purely in go.
I will probably experiement a bit with this at some point: https://magefile.org/
The only real hurdle for now, is how it discovers commands stored in plans.
TestConfig often fails and then succeeds on the next run.
It does not make sense that scripts do not have actions, so fail running the script if so. Plus maybe consider a warning when doing an overall validation.
Would be great with a chapter in the readme on how to install shuttle. Should be easy to do since we are using goreleaser to generate the releases (others must have written mostly the same stuff)
When a run
script is executed with a exit code other than 0, it's outputted wrong:
shuttle failed
Failed executing shell script `<shell command>`
Exit code: %!(EXTRA int=1)
This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.
These updates are awaiting their schedule. Click on a checkbox to get an update now.
.github/workflows/build.yml
actions/checkout v3
actions/setup-go v3
actions/cache v3
.github/workflows/release-drafter.yml
release-drafter/release-drafter v5
.github/workflows/release.yml
actions/checkout v3
actions/setup-go v3
actions/cache v3
goreleaser/goreleaser-action v3
go.mod
github.com/Masterminds/sprig/v3 v3.2.2
github.com/cli/safeexec v1.0.0
github.com/go-cmd/cmd v1.4.1
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510@e7afc7fbc510
github.com/pkg/errors v0.9.1
github.com/spf13/cobra v1.5.0
github.com/stretchr/testify v1.8.0
gopkg.in/yaml.v2 v2.4.0
You may have used git extensions before without knowing. Those are the commands you run after git, such as git ignore
, or other third party git plugins. This is a useful way of using the git name, to namespace the cli tool.
The way those work is that git, will look for binaries in PATH
on your machine, if it finds a binary with the prefix, git-
, it will allow those to be called. Such as git-ignore
can now be called with git ignore
(notice the lack of -
).
git ignore -> git -> git-ignore
. Or for shuttle
shuttle <some-binary> -> shuttle -> shuttle-<some-binary>
replace with whatever tool you'd like to expose.
shuttle would also just forward all commandline arguments, such as.
shuttle release --env dev -> shuttle-release --env dev
shuttle
Right now shuttle is built to handle code sharing between projects often in a multi-repository setting. This feature would allow us to ship additional features, which may make shuttle useful outside of project work. This could be everything from working with other third party software, such as github actions, renovate, reviewing code. Specifically tools which aren't used by one specific project, but goes across a developers
workflow on said projects.
If we wanted to, we could even do the same for hamctl, such that we would have a shuttle-release
, which would just point at hamctl
for actions. This would group all of our tools under one umbrella, without having to add specific code to shuttle itself.
Let me know what you think!
We could simply do the same as git, and look for binaries in PATH
, if found we add an entry to help menu, such that they show up when using shuttle -h
.
This would require looking in all folders mentioned in $PATH
, or just a specific location we decide for shuttle
, and if the binary is in there we pick it up and make it available.
As a user, I would like to architect my scripts in smaller chunks, but be able to call them all with a "super target", so that I can make only one call to shuttle.
Example (written in make syntax)
# Super target for destroying infra.
.PHONY: all-destroy
all-destroy: vacuum-cluster \
cluster-rm \
networking-rm \
vpc-rm \
If a shuttle has_script integration
, then we could easily extend plan with additional functionality
Shuttle telemetry doesn't react to an old lock file, or clean up after itself
It would be great if one could get the description of a specific script and it's arguments like below.
If the available arguments could be displayed as with the build in commands (shuttle run --help
) it would be even better.
shuttle run build --help
Support scripts with default values for non-required arguments
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.