mintel / build-harness Goto Github PK

View Code? Open in Web Editor NEW

This project forked from charlieparkes/build-harness

2.0 2.0 6.0 503 KB

🤖Collection of Makefiles to facilitate building Python/Golang projects, Dockerfiles, and more

License: Apache License 2.0

Dockerfile 2.54% Makefile 76.19% Shell 20.39% Python 0.88%

build-harness's People

Contributors

Stargazers

Watchers

Forkers

awichmann-mintel ekishchukova aspage2 jtdoepke liamcato dmitchell-mintel

build-harness's Issues

Should isort be included as default lint target?

isort seems having compatibility issues with black psf/black#333 PyCQA/isort#694

There's a certain config to work around the issue, and implemented here through #27

I want to start this as dicussion of is isort provide enough value that we want to take on the maintenance responsibility for the incapability and potential breaking change in the future?

Run isort before black

Auto-init build harness without running bh/init

make dist depends on pipenv but the messaging is kinda bad

This is what happens if you use the python/dist target without having pipenv installed.

awichmann@CHI-LX-L-018 '15:26:27' (upgrade_to_build_harness) '~/Code/everest_code/libraries/workflow-client'
> $ make dist
Writing requirements from Pipfile to requirements.txt and requirements-dev.txt
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'pipenv'
/home/awichmann/Code/everest_code/libraries/build-harness/modules/pipenv/Makefile.env:56: recipe for target 'pipenv/generate_requirements' failed
make: *** [pipenv/generate_requirements] Error 1

It would be nice if build-harness first checked for pipenv's installation and alerted the user with a message like "your package needs pipenv installed as a dev-dependency" instead of printing this stacktrace.

Automatically install required python packages in a virtualenv if not available on the system

Compose commands crash (after running succesfully?) as if docker-compose was called incorrectly

Creating svc_1966417_501_gateway-cache_1 ... done
Creating svc_1966417_501_gateway-db_1    ... done
Creating svc_1966417_501_gateway_1       ... done
Define and run multi-container applications with Docker.

Usage:
  docker-compose [-f <arg>...] [options] [COMMAND] [ARGS...]
  docker-compose -h|--help

Options:
  -f, --file FILE             Specify an alternate compose file
                              (default: docker-compose.yml)
  -p, --project-name NAME     Specify an alternate project name
                              (default: directory name)
  --verbose                   Show more output
  --log-level LEVEL           Set log level (DEBUG, INFO, WARNING, ERROR, CRITICAL)
  --no-ansi                   Do not print ANSI control characters
  -v, --version               Print version and exit
  -H, --host HOST             Daemon socket to connect to

  --tls                       Use TLS; implied by --tlsverify
  --tlscacert CA_PATH         Trust certs signed only by this CA
  --tlscert CLIENT_CERT_PATH  Path to TLS certificate file
  --tlskey TLS_KEY_PATH       Path to TLS key file
  --tlsverify                 Use TLS and verify the remote
  --skip-hostname-check       Don't check the daemon's hostname against the
                              name specified in the client certificate
  --project-directory PATH    Specify an alternate working directory
                              (default: the path of the Compose file)
  --compatibility             If set, Compose will attempt to convert keys
                              in v3 files to their non-Swarm equivalent
  --env-file PATH             Specify an alternate environment file

Commands:
  build              Build or rebuild services
  config             Validate and view the Compose file
  create             Create services
  down               Stop and remove containers, networks, images, and volumes
  events             Receive real time events from containers
  exec               Execute a command in a running container
  help               Get help on a command
  images             List images
  kill               Kill containers
  logs               View output from containers
  pause              Pause services
  port               Print the public port for a port binding
  ps                 List containers
  pull               Pull service images
  push               Push service images
  restart            Restart services
  rm                 Remove stopped containers
  run                Run a one-off command
  scale              Set number of containers for a service
  start              Start services
  stop               Stop services
  top                Display the running processes
  unpause            Unpause services
  up                 Create and start containers
  version            Show the Docker-Compose version information
make[1]: *** [compose/up] Error 1
make: *** [up] Error 2

Lint should include unused import detection

Follow the same logic as https://prettier.io/docs/en/comparison.html lint should cover both formatting and code quality check. Currently black and isort provide the check of code formating, but the code quality check for issues like unused import is missing

Add container-scanner

I would like to integrate https://github.com/aquasecurity/trivy

We use this as part of our CI/CD pipelines on Gitlab.

https://gitlab.com/mintel/satoshi/ci-cd/pipeline/-/blob/master/commands/container_scan

A few important notes:

We support all the trivy env-vars
We support the .trivyignore file
We support our shared CVE whitelist (which may not be appropriate for build-harness)

Add sast-scanner

See:

We're starting to integrate SAST into gitlab-ci pipelines, but we should also be able to run locally too.

Merge in upstream, and understand what upstream we should refer to

The diff between mintel/alphachai repos is minimal, but the diff between mintel/cloudposse repos is quite large :)

I would prefer the following:

Fork from cloudposse, since that's going to be getting regular changes
Maintain build-harness in such a way that bringing in upstream changes has minimal (or no) conflicts. If we're always having to deal with conflicts, then no one is going to want to maintain this :)

@alphachai (hope this still works!). We're starting a few new projects soon which will want shared Mkefiles, so hoping to put work into build-harness here. We want to figure out ownership first (probably SRE+some devs?).

Keen to get your thoughts on this though - there's no impact if we re-fork against cloudposse right? If end up creating useful PR's into build-harness (even the mintel repo), then you could still pull those in.

Also, was there any advantage in removing un-used modules such as #59 - I feel that removing stuff may lead to conflicts when bringing in upstream changes.

Support running (most) commands against docker image

I'm trying to promote running most of our development commands against a docker image.

Typically this would cover

Running the site
Linting
Functional Tests
Unit Tests
e2e Tests
Visual Regression Tests
Coverage
Container Scanning
SAST Scanning
Pipfile locking
Build docs

We shouldn't require python/pipenv/pytest/python etc to be installed locally to be able to develop an app :)

Target for auto-constructing .env based on a env.template.json

yaml would be better, but I want to rely on standard python only until I get magical package management handled

Add commands for Yarn / Lerna

Fix CI (and switch to Github actions?)

I've noticed the CI is broken (see #65 )

This might be a good opportunity to review the CI, and also switch to Github actions - main reason for this is that no one is really managing the Travis/Circle integrations from a user-management/accounts point of view.

Github actions would at least get rid of this problem, and I suspect over time integrate nicer with Github's UI.

We use it on some of our Github repos already (SRE team is gradually migration towards it and away from CircleCI for public repos).

https://docs.github.com/en/free-pro-team@latest/actions

cd /home/travis/build/alphachai/boto3-fixtures/dummy_lambda/dist && zip build.zip pyproject.toml

	zip warning: name not matched: pyproject.toml

zip error: Nothing to do! (build.zip)

/home/travis/build/alphachai/boto3-fixtures/build-harness/modules/faas/Makefile:38: recipe for target 'faas/build/python/zip' failed

make[1]: *** [faas/build/python/zip] Error 12

Lack of a pyproject.toml breaks faas/build/python/zip.