weaveworks / build-tools Goto Github PK

View Code? Open in Web Editor NEW

45.0 52.0 20.0 422 KB

Collection of build & test tools shared by various Weaveworks projects

License: Other

Shell 53.61% Go 11.37% Makefile 2.88% Python 8.92% HCL 17.89% Dockerfile 3.24% Starlark 2.09%

build-tools's Introduction

Weaveworks Build Tools

Included in this repo are tools shared by weave.git and scope.git. They include

bazel-rules: Bazel build rules used in our projects
build: a set of docker base-images for building weave projects. These should be used instead of giving each project its own build image.
provisioning: a set of Terraform scripts to provision virtual machines in GCP, AWS or Digital Ocean.
config_management: a set of Ansible playbooks to configure virtual machines for development, testing, etc.
cover: a tool which merges overlapping coverage reports generated by go test
files-with-type: a tool to search directories for files of a given MIME type
lint: a script to lint go, sh and hcl files; runs various tools like golint, go vet, errcheck, shellcheck etc
rebuild-image: a script to rebuild docker images when their input files change; useful when you using docker images to build your software, but you don't want to build the image every time.
shell-lint: a script to lint multiple shell files with shellcheck
socks: a simple, dockerised SOCKS proxy for getting your laptop onto the Weave network
test: a script to run all go unit tests in subdirectories, gather the coverage results, and merge them into a single report.
runner: a tool for running tests in parallel; given each test is suffixed with the number of hosts it requires, and the hosts available are contained in the environment variable HOSTS, the tool will run tests in parallel, on different hosts.
scheduler: an appengine application that can be used to distribute tests across different shards in CircleCI.

Requirements

lint requires shfmt to lint sh files; get shfmt with

curl -fsSLo shfmt https://github.com/mvdan/sh/releases/download/v1.3.0/shfmt_v1.3.0_linux_amd64
chmod +x shfmt

(we pin that version, and it doesn't build from the source repo any more)

Using build-tools.git

To allow you to tie your code to a specific version of build-tools.git, such that future changes don't break you, we recommendation that you git subtree this repository into your own repository:

git subtree add --prefix tools https://github.com/weaveworks/build-tools.git master --squash

To update the code in build-tools.git, the process is therefore:

PR into build-tools.git, go through normal review process etc.
Do git subtree pull --prefix tools https://github.com/weaveworks/build-tools.git master --squash in your repo, and PR that.

Getting Help

If you have any questions about, feedback for or problems with build-tools:

Invite yourself to the Weave Users Slack.
Ask a question on the #general slack channel.
File an issue.

Weaveworks follows the CNCF Code of Conduct. Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting a Weaveworks project maintainer, or Alexis Richardson ([email protected]).

Your feedback is always welcome!

build-tools's People

Contributors

Stargazers

Watchers

Forkers

n054 kinvolk-archives alexxnica kryndex meghalidhoble tomwilkie m0rganic ik-automation murali-reddy morristech dinosk isabella232 forkdown ekmixon alexrogalskiy zaxebo1 natefaerber

build-tools's Issues

Updating subtree-d code should be automated

Otherwise we never do it properly

tools/lint fails own check

shfmt v1.0.0 changes = to ==:

# git diff
diff --git a/tools/lint b/tools/lint
index 15a97d4..566e329 100755
--- a/tools/lint
+++ b/tools/lint
@@ -173,7 +173,7 @@ lint() {
         coverage.html) return ;;
     esac
 
-    if [[ "$(file --mime-type "${filename}" | awk '{print $2}')" = "text/x-shellscript" ]]; then
+    if [[ "$(file --mime-type "${filename}" | awk '{print $2}')" == "text/x-shellscript" ]]; then
         ext="sh"
     fi
 
# shfmt --version
v1.0.0

This breaks the weaveworks/service-conf build

Output test results in JUnit XML format

So CircleCI can parse them. I don't know how useful this would be, but it seems fairly straighforward.

I think this would involve extending the summary function in runner/runner.go

Some guidance to the desired format here

Rebuild-with-ssh on failed builds cannot create GCE instances

If the tests have failed, the scheduler GC keeps deleting the instances, because the build status is listed as timedout (or w/e).

To tell when the ssh-enabled build is cancelled/timedout, we should also check for "why": "ssh", and maybe something else.

API Call for reference: https://circleci.com/api/v1/project/weaveworks/weave

push-images doesn't fail when a push fails

Stop pre-building Go std library

These lines:

build-tools/build/golang/Dockerfile

Lines 25 to 27 in dedde7f

 RUN go clean -i net && \ 

 go install -tags netgo std && \ 

 go install -race -tags netgo std

are not necessary since Go added a per-target build cache, and probably don't achieve anything
since we typically bind-mount another directory for the build cache when compilations are run: https://github.com/weaveworks/prom-aggregation-gateway/blob/c4415bbe1411d77eb4132f3df51c87cf7fcca840/Makefile#L60

Related: #40

Port ./scheduler to python3

Since python2 is EOL, we should port ./scheduler to python3.

Vendor Go dependencies

To prevent things like #38 from breaking builds again

Include build-tools in build image

My understanding is:

the base build images provide the basic environment we need for a "standard" weaveworks build
the build-tools repo provides the basic tools we need for same

However, the tools in build-tools can depend on specific versions of things. Often, spellcheckers and linters introduce new checks in new versions, breaking existing code.

It would be ideal if the build-tools tools could be provided as standard in the build images. That way, we wouldn't need to subtree this repo in, and we could be sure that the dependencies were the right ones for the tools.

Add a yaml linter

since I managed to waste a couple of hours not spotting I had an extra space, which showed up as a "did not find expected key" response from kubectl

Python linting & formatting

flake8 and https://github.com/google/yapf

Current build fails

Upon merging #129 the build failed:

...
# cd .; git clone https://go.googlesource.com/lint /go/src/golang.org/x/lint
Cloning into '/go/src/golang.org/x/lint'...
fatal: remote error: Git repository not found
package golang.org/x/lint: exit status 128
The command '/bin/sh -c go get -tags netgo 		github.com/FiloSottile/gvt 		github.com/client9/misspell/cmd/misspell 		github.com/fatih/hclfmt 		github.com/fzipp/gocyclo 		github.com/gogo/protobuf/gogoproto 		github.com/gogo/protobuf/protoc-gen-gogoslick 		github.com/golang/dep/... 		github.com/golang/lint/golint 		github.com/golang/protobuf/protoc-gen-go 		github.com/kisielk/errcheck 		github.com/mjibson/esc 		github.com/prometheus/prometheus/cmd/promtool && 		rm -rf /go/pkg /go/src' returned a non-zero code: 1
make: *** [golang/.uptodate] Error 1
....

CI test output shown twice

Take a look at a recent Weave CI test, in particular the output produced by the

$ cd $SRCDIR/test; eval $(./gce.sh hosts); export COVERAGE=true; ./run_all.sh (0)

test step.

It contains the entire output twice. Careful examination shows that it is literally copied, i.e. the tests have in fact only been run once - otherwise reported timing would differ.

The same happens in Scope CI.

I've gone back all the way to our first successful CI run, and the same problem is visible there.

So I do wonder whether this is in fact a recent Circle CI display breakage.

test runner can lose individual result messages

Example: there are 25 tests run, but only 23 of the "finished with success" messages.

>>> Running ./800_plugin_cross_hosts_2_test.sh on [host1-4339-0.us-central1-a.positive-cocoa-90213 host2-4339-0.us-central1-a.positive-cocoa-90213]
>>> Running ./190_no_fastdp_2_test.sh on [host3-4339-0.us-central1-a.positive-cocoa-90213 host4-4339-0.us-central1-a.positive-cocoa-90213]
>>> Running ./685_proxy_weave_run_test.sh on [host5-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./685_proxy_weave_run_test.sh finished with success after 7.4 secs
>>> Running ./680_proxy_hostname_derivation_test.sh on [host5-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./800_plugin_cross_hosts_2_test.sh finished with success after 11.6 secs
>>> Running ./165_reset_2_test.sh on [host1-4339-0.us-central1-a.positive-cocoa-90213 host2-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./190_no_fastdp_2_test.sh finished with success after 17.7 secs
>>> Running ./150_connect_forget_2_test.sh on [host3-4339-0.us-central1-a.positive-cocoa-90213 host4-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./165_reset_2_test.sh finished with success after 19.6 secs
>>> Running ./105_frag_2_test.sh on [host1-4339-0.us-central1-a.positive-cocoa-90213 host2-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./150_connect_forget_2_test.sh finished with success after 19.8 secs
>>> Running ./100_cross_hosts_2_test.sh on [host3-4339-0.us-central1-a.positive-cocoa-90213 host4-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./680_proxy_hostname_derivation_test.sh finished with success after 35.6 secs
>>> Running ./670_proxy_tls_test.sh on [host5-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./670_proxy_tls_test.sh finished with success after 6.7 secs
>>> Running ./666_abuse_of_start_test.sh on [host5-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./100_cross_hosts_2_test.sh finished with success after 22.3 secs
>>> Running ./655_proxy_large_http_chunks_test.sh on [host3-4339-0.us-central1-a.positive-cocoa-90213]
>>> Running ./640_proxy_restart_reattaches_test.sh on [host4-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./666_abuse_of_start_test.sh finished with success after 11.6 secs
>>> Running ./630_proxy_dns_test.sh on [host5-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./655_proxy_large_http_chunks_test.sh finished with success after 6.1 secs
>>> Running ./620_proxy_entrypoint_command_test.sh on [host3-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./640_proxy_restart_reattaches_test.sh finished with success after 16.4 secs
>>> Running ./604_proxy_docker_py_test.sh on [host4-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./620_proxy_entrypoint_command_test.sh finished with success after 11.5 secs
>>> Running ./601_proxy_docker_py_test.sh on [host3-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./630_proxy_dns_test.sh finished with success after 18.2 secs
>>> Running ./500_weave_multi_cidr_test.sh on [host5-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./105_frag_2_test.sh finished with success after 50.4 secs
>>> Running ./295_dns_large_response_test.sh on [host1-4339-0.us-central1-a.positive-cocoa-90213]
>>> Running ./280_with_or_without_dns_test.sh on [host2-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./295_dns_large_response_test.sh finished with success after 9.0 secs
>>> Running ./270_use_name_as_hostname_test.sh on [host1-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./280_with_or_without_dns_test.sh finished with success after 10.3 secs
>>> Running ./265_dns_reverse_test.sh on [host2-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./265_dns_reverse_test.sh finished with success after 13.1 secs
>>> Running ./250_dns_negative_test.sh on [host2-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./250_dns_negative_test.sh finished with success after 7.3 secs
>>> Running ./225_dns_subdomain_test.sh on [host2-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./270_use_name_as_hostname_test.sh finished with success after 21.7 secs
>>> Running ./195_no_multicast_route_1_test.sh on [host1-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./500_weave_multi_cidr_test.sh finished with success after 34.6 secs
>>> Running ./170_ip_reuse_test.sh on [host5-4339-0.us-central1-a.positive-cocoa-90213]
>>> Test ./225_dns_subdomain_test.sh finished with success after 11.0 secs
>>> Test ./170_ip_reuse_test.sh finished with success after 33.8 secs
>>> Test ./604_proxy_docker_py_test.sh finished with success after 81.8 secs
>>> Ran 25 tests, all succeeded

Move off of gvt

#177 (review)

... At some point (in another PR) we should also get rid of gvt since it has being obsoleted by go modules.

Update wks -> wksctl circleci project name

There are errors on the app engine dashboard that look like this:

RuntimeError: Failed to get running builds for repository "weaveworks/wks". URL: https://circleci.com/api/v1/project/weaveworks/wks?circle-token=, Status code: 404. Response: {"message":"Project not found"}

The integration test runs have been moved from wks -> wksctl

Stop apt daily job running on test machines

Similar to #108 but this time do it in the right place.

This should fix many of the test flakes on 'defunct' processes

Enforce Dockerfile labels with lint

We often have the problem that we don't know where the source code for an image lives.

There's a more-or-less standard way of providing this information, using Docker labels. We should do that.

e.g. https://github.com/weaveworks/weave/blob/master/prog/weaver/Dockerfile.template#L7-L13

LABEL maintainer "Weaveworks Inc <[email protected]>"
LABEL \
      org.label-schema.name="Weave Net" \
      org.label-schema.description="Weave Net creates a virtual network that connects Docker containers across multiple hosts and enables their automatic discovery" \
      org.label-schema.url="https://weave.works" \
      org.label-schema.vcs-url="https://github.com/weaveworks/weave" \
      org.label-schema.vendor="Weaveworks"

Find a new home for the socks proxy.

Shell lint doesn't fail when it should

See https://circleci.com/gh/weaveworks/service-conf/1593?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link

./tools/lint .

ansible/ansible-helper: run shfmt -i 4 -w ansible/ansible-helper

ansible/ansible-playbook-helper: run shfmt -i 4 -w ansible/ansible-playbook-helper

ansible/playbooks/roles/bootstrap/files/ansible_prereqs.sh: run shfmt -i 4 -w ansible/playbooks/roles/bootstrap/files/ansible_prereqs.sh

ansible/update_inventory: run shfmt -i 4 -w ansible/update_inventory

ansible/update_kubernetes_certs: run shfmt -i 4 -w ansible/update_kubernetes_certs

ansible/update_master_unschedulable: run shfmt -i 4 -w ansible/update_master_unschedulable

infra/common.sh: run shfmt -i 4 -w infra/common.sh

infra/ssh-master: run shfmt -i 4 -w infra/ssh-master

infra/ssh-minion: run shfmt -i 4 -w infra/ssh-minion

Got a bunch of errors, but exit code was 0.

Distribute build tools as a single binary

Background

build-tools is a collection of shell, Python, Go, and Terraform scripts that represent build tooling common to Weaveworks projects, proprietary and open source.

It is currently embedded in each of our projects via git subtree or git submodules.

Problems

Distribution

Both git subtree and git submodules are problematic. They are unfamiliar commands to most git users, which introduces a level of friction with both new & existing hires.

When we make an improvement to build-tools, we face the problem of manually identifying all repositories which use it and then filing a PR one by one. This becomes more and more difficult the more we expand.

Language

Writing robust shell code is hard, even with shellcheck. Writing shell code that communicates intent is even harder.

Solution

Rewrite everything in Go
Have a single, top-level binary wbuild with subcommands that make all of the required build and lint functionality available
Continuously release this binary to GitHub
Update all projects that use build-tools to
- remove the subtrees
- download the latest binary as part of their CI

Pinning

While projects should just fetch the latest build-tools, we should ensure that each master commit has a downloadable binary so that projects that need to pin to older versions can.

Things that cannot be written in Go

Provisioning

We have a collection of Terraform files for provisioning test machines.

This should be moved to Weave Net repository. If making it public is impractical, then it should move to its own private repository.

Add staticcheck

As used in weaveworks/weave#2843

I expect some tuning would be required.

Take out build.sh chmod

These lines:

build-tools/build/golang/build.sh

Lines 12 to 20 in dedde7f

 # If we run make directly, any files created on the bind mount 

 # will have awkward ownership. So we switch to a user with the 

 # same user and group IDs as source directory. We have to set a 

 # few things up so that sudo works without complaining later on. 

 uid=$(stat --format="%u" "$SRC_PATH") 

 gid=$(stat --format="%g" "$SRC_PATH") 

 echo "weave:x:$uid:$gid::$SRC_PATH:/bin/sh" >>/etc/passwd 

 echo "weave:*:::::::" >>/etc/shadow 

 echo "weave ALL=(ALL) NOPASSWD: ALL" >>/etc/sudoers

can be removed if we run the build as the desired user. For example see weaveworks/scope@833f947

Stop using docker's mflag package

It was removed upstream in moby/moby#25354

We should probably use cobra instead, just like docker did.

Replace individual Go checking tools with golangci-lint

Many of these files: