This repository contains a set of scripts and configuration files that can be used to deploy Buildbarn on various platforms. Buildbarn is pretty flexible, in that it can both be used for single-node remote caching setups and large-scale remote execution setups. Unless noted otherwise, the configurations in this repository all use assume the following setup:

• Sharded storage, using the Buildbarn storage daemon. To apply the sharding to client RPCs, a separate set of stateless frontend servers is used to fan out requests.
• Remote execution of build actions, using container images from Act, distributed by catthehacker, used for running GitHub Actions locally under Ubuntu 22.04.
• An installation of the Buildbarn Browser.

Below is a diagram of what this setup Buildbarn looks like. In this diagram, the arrows represent the direction in which network connections are established.

This example aims to showcase a very simple build and test with remote execution using docker-compose as the deployment for Buildbarn. We will be compiling examples from the abseil-hello project using Bazel.

First clone the repo and start up a docker-compose example:

git clone https://github.com/buildbarn/bb-deployments.git
cd bb-deployments/docker-compose
./run.sh

You may see initially see an error message along the lines of:

worker-ubuntu22-04_1  | xxxx/xx/xx xx:xx:xx rpc error: code = Unavailable desc = Failed to ...: connection error: desc = "transport: Error while dialing dial tcp xxx.xx.x.x:xxxx: connect: connection refused"

This is usually because container of the worker has started before the scheduler or runner and so it cannot connect to them. After a second or so, this error message should stop.

Bazel can perform remote builds against these deployments by using toolchains adapted to the remote environment. The script tools/remote-toolchains/extract-bazel-auto-toolchains.sh has been used to construct such a C++ toolchain which is activated using --config=remote-ubuntu-22-04, see .bazelrc and WORKSPACE for the exact setup.

Note that the name and SHA of the container image is configured in multiple places: BUILD.bazel, .jsonnet configuration for Buildbarn and for the actual runner container (docker compose, kubernetes yaml, etc...)

Now try a build (using bazel or bazelisk):

bazel build --config=remote-ubuntu-22-04 @abseil-hello//:hello_main

The output should look something like:

INFO: 33 processes: 4 internal, 29 remote.
INFO: Build completed successfully, 33 total actions

You can check to see if the binary has built successfully by trying:

bazel run --config=remote-ubuntu-22-04 @abseil-hello//:hello_main

You may experience problems with wrong version of glibc compared to what the remote is building for.

Equally, you can try to execute a test remotely:

bazel test --config=remote-ubuntu-22-04 @abseil-hello//:hello_test

Which will give you an output containing something like:

INFO: 49 processes: 4 internal, 45 remote.
INFO: Build completed successfully, 49 total actions
@abseil-hello//:hello_test                                     PASSED in 0.1s

Executed 1 out of 1 test: 1 test passes.

You might experience problems loading abseil-hellos/libhello.so, in which case you can link statically by using --dynamic_mode=off.

Next, we will try out the remote caching capability. If you clean your local build cache and then rerun a build:

bazel clean
bazel build --config=remote-ubuntu-22-04 @abseil-hello//:hello_main

You'll see an output containing information that we hit the remote cache instead of executing on a worker.