cga1123 / slugcmplr Goto Github PK

🐌 A CLI for compiling your Heroku apps into slugs.

License: MIT License

Go 98.48% Shell 1.52%

heroku slug compiler build-tools build-automation continuous-deployment works-with-codespaces

slugcmplr's Introduction

slugcmplr

slugcmplr allows you to compile Heroku compatible slugs using official and custom buildpacks, in the same manner as the Heroku Slug Compiler

This enables more control over the build process and allows for detaching building and releasing your Heroku applications.

Install

With the go toolchain installed:

# In order to use this as a library
go get github.com/cga1123/slugcmplr

# In order to install as a CLI
go install github.com/cga1123/slugcmplr@latest

There are also precompiled binaries and their associated checksums are available attached to tagged releases.

Info

slugcmplr has 3 steps/sub-commands:

`prepare [APPLICATION] --build-dir [BUILD-DIR] --source-dir [SOURCE-DIR]`

In the prepare step, slugcmplr will fetch the metadata required to compile your application. It will copy your project SOURCE-DIR into BUILD-DIR/app.

It will fetch the buildpacks as defined by your Heroku application, download, and decompress them into BUILD-DIR/buildpacks. If using official buildpacks (e.g. heroku/go, heroku/ruby) slugcmplr will use the same buildpack as currently deployed to Heroku's production environment.

It will fetch the config vars as defined by your Heroku application and put them into the BUILD-DIR/environment directory.

Finally, prepare writes metadata (such as the application name, stack, buildpack order, source version) to the BUILD-DIR/compile.json file to allow the compile step to bootstrap itself.

`compile --build-dir [BUILD-DIR] --cache-dir [CACHE-DIR]`

In the compile step, slugcmplr executes your buildpacks in the specified order, outputs your Heroku slug, and uploads it to Heroku for future release.

compile will output the slug for your application to BUILD-DIR/app.tgz

compile will output metadata about the compilation to BUILD-DIR/release.tgz, this contains information such as the slug ID as uploaded to Heroku.

The CACHE-DIR will be used by the buildpacks as their cache argument to speed up builds in the future, as per the Buildpack API

To guarantee full compatibility, it is recommended to run this step using Heroku's build containers. heroku/heroku:20-build or heroku/heroku:18-build.

`release --build-dir [BUILD-DIR]`

In the release step, slugcmplr triggers a release of your previously compiled slug.

It uses the BUILD-DIR/release.json file in order to fetch metadata in order to create this release.

You can optionally pass --app [APPLICATION] to target an application that is different from the one you built from. This will work as long as the applications are in the same Heroku team. (This is becuase the slug must be accessible to the application).

You can optionally pass --commit [COMMIT] to associate this release with a separate commit from the one used to build this slug initially.

Authentication

The slugcmplr CLI looks for credentials to api.heroku.com in your .netrc file, this is the same technique used by heroku/cli and so if you are currently making use of the heroku command during CI, you should already be logging in somehow and have no issues.

Otherwise populating your .netrc should be a case of adding something equivalent to the following script (assuming the HEROKU_EMAIL and HEROKU_API_KEY are correctly populated environment variables):

cat << EOF >> ${HOME}/.netrc
machine api.heroku.com
  login ${HEROKU_EMAIL}
  password ${HEROKU_API_KEY}
EOF

By default, slugcmplr will look in ${HOME}/.netrc for the credentials, however it will respect the ${NETRC} environment variable if set and non-empty.

Testing

The majority of tests for this project are acceptance tests that will create and release live Heroku applications. These require the correct credentials to be set in the environment as well as setting a sentinel value to execute the acceptance tests:

SLUGCMPLR_ACC=true \
  SLUGCMPLR_ACC_HEROKU_PASS=<HEROKU-API-KEY> \
  SLUGCMPLR_ACC_HEROKU_EMAIL=<HEROKU-EMAIL> \
  go test -v

Fixture application are hosted in separate repositories which will be cloned and created by using the withHarness function. Fixture repositories are expected to contain a app.json file which describes the Heroku applications.

See app.json Schema

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/CGA1123/slugcmplr

See issues if you want any inspiration as to what to help with or of any pending discussion/work.

This project is Codespaces compatible! If you want to get started quickly spin one up.

The script directory should help you get going with building and testing.

Some buildpacks are not compatible with arm/arm64 architectures, you may experience issues testing slugcmplr on an M1 Chip, even if using docker containers.

For more background on this you might find this Medium article helpful.

slugcmplr's People

Contributors

Stargazers

Watchers

Forkers

lucia-golovin

slugcmplr's Issues

Improve reliability with retries

Sometimes the network is flaky, slugcmplr should handle common network errors by retrying.

5XX
Connection resets
etc...

Should all be retryable operations.

This can be especially annoying if the network is flaky after compilation (during upload).

Heroku Client
Slug Upload Client
Buildpack Download

Use Procfile to determine image command

Adding a --process options to slugcmplr image might be nice as a way to avoid having to write out a full command.

e.g. slugcmplr image --build-dir [BUILD-DIR] --process web

The argument to --process can be used to fetch the command as registered in the builddir/app/Procfile (assuming that the Procfile hasn't been removed by any buildpack.

Add a `container` subcommand

It should be easy to arbitrarily create a containerised version of any heroku application.

If we can build the application and then mount that on a heroku/heroku image then we have a container image.

The container subcommand should run the build process and output a new container with the contents of the build-dir mounted in the /app directory of a heroku/heroku image.

Default to using an image that matches the build tag or commit (?)

The compile sub-command (when run without --local) will pick up an appropriate image for the stack (e.g ghcr.io/cga1123/slugcmplr:heroku-20).

In order to ensure backwards compatibility when release new version (especially those that might have breaking changes!) the command should probably be a little bit more picky, choosing an image that matches the tag/build version of the locally invoking binary.

goreleaser

Use goreleaser to automagically build and release compiled artifacts when a new tag is pushed.

This can make it easier to install the slugcmplr binary without requires the go toolchain.

See: goreleaser

Support symlinks in application slug

The current implementation of targz which is used to create the final slug tarball doesn't currently support adding symlinks.

It should be updated to do that, allowing only for symlinks within the slug/application directory.

Failure to detect should fail compilation

If a buildpack fails to detect, the compilation should also fail (rather than skipping the buildpack)

This should be in line with Heroku's behaviour when building (need to check that).

Inject logging and timing helpers into context

It might be nice to be able to collect logs, timings, events, via some interface that could be injected into the context for different commands.

Maybe see if there's a nice way to do it using open telemetry?

Support `.slugignore`

.slugignore files can be used to nuke a bunch of files/folders from the final slug, making it smaller.

See: https://devcenter.heroku.com/articles/slug-compiler#ignoring-files-with-slugignore

Should be feasible by using the Skip options in https://pkg.go.dev/github.com/otiai10/copy#Options

Create `slugcmplr` package

To facilitate the use of slugcmplr as a library rather than only as a cli, the core (non-cli) logic and structs should be exposed in their own package.

This means people can easily import slugcmplr into their own projects and modify the logic as they see fit.

The CLI logic can be moved into cmd/slugcmplr or cli/slugcmplr

Allow a `timeout` options on builds

At least the compile command should have a --timeout option available to kill the compilations after a while.

e.g.

--timeout 30m

This should propagate through the command context so that we can cancel appropriately.

Add a `prepare` + `compile` command

prepare and compile are separate commands at the moment, should there be a combined one so it's a fire and forget job?

slugcmplr run [APP] --build-dir [BUILD-DIR] --source-dir [SOURCE-DIR] --cache-dir [CACHE-dir]

Flaky Acceptance Test

See: https://github.com/CGA1123/slugcmplr/runs/3744106356

Looks like there might be a flake in this test!

Is there a way to fix this or should some retry logic be implemented?

Support non-github git refs

See: https://devcenter.heroku.com/articles/buildpacks#buildpack-references

Would be nice to support generic git references as buildpack URLs, currently only support github.com references (because the tarball API is very convenient)

Using go-git to checkout a particular version should be not too difficult.

Should be a shallow clone + only checkout the ref required.

Might be nice to support using .netrc again for auth?

Compile applications locally

It shouldn't be horribly difficult to run the build process locally as part of a GitHub Action/CircleCI (CI provider X) build.

Heroku has docker build images available to use on dockerhub

heroku/heroku:20-build / heroku/heroku:18-build

The Buildpack API is fairly well documented (there are some specifics around caching that would probably need to be clarified/investigated, think the cache is per app, not per-buildpack)

However the high-level would be:

Fetch all the environment variables for an application
Dump them into a folder, filename is the variable name, contents are the contents
Overwrite some build specific env vars that Heroku provides (like the git SHA and stuff)
Clone all the buildpacks
Clone the source code (or fetch it somehow else)
Mount source + buildpacks into the docker image
Run the buildpacks in order (bin/detect, bin/compile, bin/release)
Need to support export scripts as well being created by buildpacks.
- Think this means exec'ing the export script after a buildpack is done to make stuff available available for other buildpacks down the chain.
Tar up the result (?)
That's it?

This would mean fully detaching the build process from Heroku's infrastructure itself (apart from fetching the list of buildpacks, config vars, application features, and stack) which might be nice and remove the need for a "ghost"/"compile" application being managed and synchronised.

Ensure `build-in-app-dir` behaviour is the default behaviour

Cleanup output

The output is a little bit messy, might be nice to clean it up?

More fixture applications

It would be nice to have at least 1 fixture application to smoke test per available official buildpack.

It might also be good to have a healthcheck as part of that smoke test?

i.e. expect every app to return a 200 OK (with retries?) on the / endpoint at least to verify a clean deploy with a process that can actually start.

This might need to be lenient delayed for slow web servers that take a while to connect to ${PORT}