buchgr / bazel-remote Goto Github PK

When looking up Action messages stored in the ActionCache, the results are only usable if all the referenced output blobs are also available in the CAS. When the client performs a GET or HEAD request, bazel-remote should confirm that these blobs exist (thereby moving them to the "safe" end of the LRU queue), and otherwise return 404 (even if the Action exists).

Since adding extra validation might break existing users (eg if using bazel-remote's ActionCache as a generic blob store), perhaps we should add a new raw/ REST API endpoint that behaves like ac/ does currently (ie no validation). I'm not sure if there's a reasonable automatic migration path, but sysadmins can at least rename preexisting ac/ directories to raw/ before upgrading bazel-remote.

Another option would be to add a command line flag to turn on extra validation for HTTP ac/.

We found that on some hardware this absolutely kills the performance of the system e.g. a Mac Pro became completely unresponsive. In general, there is no need for calling Sync() on every write as in the worst case it's ok to lose data. We should only ensure that on loading the entries from disk that there is data integrity.

We should probably not specify username and password on the command line but instead support something like the .htpasswd format that's used by web servers for basic authentication.

https://httpd.apache.org/docs/2.4/programs/htpasswd.html

Right now bazel-remote just logs to stdout/stderr, which means it needs to be spawn by redirecting those to a file and the log files managed externally.

Alternatively, it could use something like lumberjack and rotate the logs itself automatically. It would need some new configuration parameters to determine log location (the default would be under the cache directory), and how often to rotate the logs.

I propose the following new config options:

log_path: path.log   # optional, if relative path it goes under cacheDir, defaults to bazel-remote.log
log_max_size:   500  # in megabytes
log_max_backups:  3  # number of backup files to keep around
log_max_age:     28  # how many days to keep the around for
log_compress:  true  # disabled by default

I've been testing out bazel-remote and noticed that the disk usage exceeds the max_size. It seems like the larger I set the max to, the larger the error:

• max_size=5 actual disk usage of cas dir is >5.1gb
• max_size=180 actual disk usage of cas dir >182gb

This was surprising for me and results in 500 errors. In testing, I set max_size=199 where the volume /data was only~200gb in size. bazel-remote ended up returning 500 on PUT since the total usage exceeded the available space, so I'm not sure what size it would have grown to.

I think we could add support for debugging cache misses between two machines to the remote cache.

1. Start the cache in debug mode
2. It tells you to run the first build.
3. You tell it once finished.
4. It tells you to run the second build.
5. You tell it once finished.
6. It then finds the actions that had the same inputs but produced (one or more) different outputs and prints them in a human readable format (i.e. the command, the environment, the input/output file names with hashes).

This can detect and help fix three kinds of errors:

1. Find non-determinism in the build, either on the same or different machines.
2. Tell you exactly the command(s) that produced different outputs on different machines. Once one knows which commands produce different outputs it's comes down to checking if the versions of the tools match etc.

Thoughts?

cc: @nicolov @jgavris @BenTheElder

Is there any reason the cache directory defaults to 0744 instead of the more common 0755 or even respecting umask?

I'm talking about this code.

This results in errors similar to:

ERROR: $WORKSPACE/$PACKAGE/BUILD:1:1: output '$PACKAGE/lib$TARGET-class.jar' was not created
ERROR: $WORKSPACE/$PACKAGE/BUILD:1:1: output '$PACKAGE/lib$TARGET.jar_manifest_proto' was not created
ERROR: $WORKSPACE/$PACKAGE/BUILD:1:1: output '$PACKAGE/lib$TARGET.jdeps' was not created
ERROR: $WORKSPACE/$PACKAGE/BUILD:1:1: output '$PACKAGE/lib$TARGET-native-header.jar' was not created
ERROR: $WORKSPACE/$PACKAGE/BUILD:1:1: not all outputs were created or valid

when using java rules for example.

I think there are some bugs in EnsureSpacer, as my disks keep filling up. Anyways, I've been reading the code and I don't understand why it uses the low/high water marks to trigger deletion. We could store and update the atimes in cache.files, and ensure that there's enough space before each upload, instead of doing it in bulk once the cache goes over the high water mark. This would ensure optimal disk usage.

Not sure if that is possible, but it would be nice to somehow protect external dependencies from being automatically deleted. The idea behind this is to be able to still enforce a maximum size for project targets but be guaranteed to have your external dependencies at all times and therefore have some protection against the case of third party dependencies having been removed or internet downtime (given the server being in the local network).

Let me know what you think of this idea.

It seems like the AC and CAS hashes could collide since they both appear to be stored in <cache>/<hash>.

Perhaps instead we could store them at <cache>/ac-<hash> and <cache>/cas-<hash>?

We don't specify a particular version, and I think the default changed from v1 to v2 when v2.0.0 was released. Let's fix this by upgrading to v2.

I recently added some ActionResult validation, including one check that seems to be technically incorrect but still useful in practice: zero-length serialized ActionResult messages are currently considered invalid.

This is technically incorrect, because an ActionResult might legitimately have no outputs apart from the exit code (and if that exit code is 0 then the serialized value could be zero-length). One possible use case for this is a sanity check. But this is bad, because a zero-length file is perhaps the most common form of data corruption, and we have no way to recognise when this is corruption and when it is legitimate.

However in practice, there are many cases where this type of ActionResult can be reasonably be expected never to occur (eg when REAPI is only used for compilation jobs, like recc or goma jobs). And in that case, maybe it's worth keeping this validation, off by default and hidden by a command line flag. Would such a flag be welcome, or should I just remove this validation?

https://github.com/bazelbuild/remote-apis

In the same way that Bazel provides binaries, bazel-remote should also provide binary releases.

This is desirable for enterprise use.

Hello,

We are investigating on using Bazel as our build system, and are assessing using bazel-remote as our remote caching. However, I am concerned that the Bazel's official documentation lists this as experimental and un-supported.

Bazel Remote Cache is an open source remote build cache that you can use on your infrastructure. It is experimental and unsupported.

Can it be confirmed that this is not experimental? I do see that there are enough commits in the past few months that it is being supported.

The cache should only garbage collect action cache entries based on last access time. CAS entries should have a reference count and only be collected if they are no longer referenced by any cache entry.

Incompatible flag --incompatible_load_python_rules_from_bzl will break Bazel Remote Cache once Bazel 1.2.1 is released.

Please see the following CI builds for more information:

• [:ubuntu: 16.04 (OpenJDK 8)](:ubuntu: 16.04 (OpenJDK 8))
• [:ubuntu: 18.04 (OpenJDK 11)](:ubuntu: 18.04 (OpenJDK 11))

Questions? Please file an issue in https://github.com/bazelbuild/continuous-integration

Important: Please do NOT modify the issue title since that might break our tools.

It's currently single-threaded and can be quite slow for large caches.

Hi, just wondering if this functionality is intended.
If I use the image you provide (buchgr/bazel-remote-cache), everything works fine.

If I built it myself, it populates /data/bazel-cache with the following:

How I build the image:

# Build image
$ bazel run :bazel-remote-image -- --norun

# Run container
$ docker run -v /data/bazel-cache:/data -p 8080:8080 bazel:bazel-remote-image

The error, described at the end, occurs when trying to build from source
machine on centos7
compiler is gcc 8.3.0 installed not local to the machine
I was needing LD_LIBRARY_PATH to pass this but this step eas not taking that into account.

I am new to bazel, I found out that option "use_default_shell_env=True" in actions.run
was necessary.
a modification of external/io_bazel_rules_go/proto/compiler.bzl in actions.run fixed the problem

bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/linux_amd64_stripped/go-protoc -protoc bazel-out/host/bin/external/com_google_protobuf/protoc -importpath google.golang.org/genproto/protobuf/api -out_path bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/proto/wkt/linux_amd64_stripped/api_go_proto%/ -plugin bazel-out/host/bin/external/com_github_golang_protobuf/protoc-gen-go/linux_amd64_stripped/protoc-gen-go -descriptor_set bazel-out/k8-fastbuild/bin/external/com_google_protobuf/source_context_proto-descriptor-set.proto.bin -descriptor_set bazel-out/k8-fastbuild/bin/external/com_google_protobuf/any_proto-descriptor-set.proto.bin -descriptor_set bazel-out/k8-fastbuild/bin/external/com_google_protobuf/type_proto-descriptor-set.proto.bin -descriptor_set bazel-out/k8-fastbuild/bin/external/com_google_protobuf/api_proto-descriptor-set.proto.bin -expected bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/proto/wkt/linux_amd64_stripped/api_go_proto%/google.golang.org/genproto/protobuf/api/api.pb.go -import 'google/protobuf/source_context.proto=google.golang.org/genproto/protobuf/source_context' -import 'google/protobuf/any.proto=github.com/golang/protobuf/ptypes/any' -import 'google/protobuf/type.proto=google.golang.org/genproto/protobuf/ptype' -import 'google/protobuf/api.proto=google.golang.org/genproto/protobuf/api' google/protobuf/api.proto)
ERROR: /net/hogsback/home1/thierry/bazel/bazel/_bazel_thierry/aa4e8273d17ebfffbcef68d297df2765/external/io_bazel_rules_go/proto/wkt/BUILD.bazel:3:1: Generating into bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/proto/wkt/linux_amd64_stripped/struct_go_proto%/github.com/golang/protobuf/ptypes/struct failed (Exit 1) go-protoc failed: error executing command
(cd /net/hogsback/home1/thierry/bazel/bazel/_bazel_thierry/aa4e8273d17ebfffbcef68d297df2765/sandbox/processwrapper-sandbox/915/execroot/main &&
exec env -
CGO_ENABLED=1
GOARCH=amd64
GOOS=linux
GOROOT=external/go_sdk
GOROOT_FINAL=GOROOT
PATH=/soft/fw/tools/gcc/gcc-8.3.0-rhel7-x64/bin:/usr/bin:/bin
bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/linux_amd64_stripped/go-protoc -protoc bazel-out/host/bin/external/com_google_protobuf/protoc -importpath github.com/golang/protobuf/ptypes/struct -out_path bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/proto/wkt/linux_amd64_stripped/struct_go_proto%/ -plugin bazel-out/host/bin/external/com_github_golang_protobuf/protoc-gen-go/linux_amd64_stripped/protoc-gen-go -descriptor_set bazel-out/k8-fastbuild/bin/external/com_google_protobuf/struct_proto-descriptor-set.proto.bin -expected bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/proto/wkt/linux_amd64_stripped/struct_go_proto%/github.com/golang/protobuf/ptypes/struct/struct.pb.go -import 'google/protobuf/struct.proto=github.com/golang/protobuf/ptypes/struct' google/protobuf/struct.proto)
Execution platform: @local_config_platform//:host

Use --sandbox_debug to see verbose messages from the sandbox
bazel-out/host/bin/external/com_google_protobuf/protoc: /lib64/libstdc++.so.6: version GLIBCXX_3.4.20' not found (required by bazel-out/host/bin/external/com_google_protobuf/protoc) bazel-out/host/bin/external/com_google_protobuf/protoc: /lib64/libstdc++.so.6: version GLIBCXX_3.4.21' not found (required by bazel-out/host/bin/external/com_google_protobuf/protoc)
bazel-out/host/bin/external/com_google_protobuf/protoc: /lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by bazel-out/host/bin/external/com_google_protobuf/protoc)
2019/12/04 19:28:24 error running protoc: exit status 1
Target //:bazel-remote failed to build
INFO: Elapsed time: 1.239s, Critical Path: 0.42s
INFO: 0 processes.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

Include a brief introduction of how to set up the development environment, what's the recommended IDE, how to run test inside IDE, etc..
I tried to set up running test inside Intellij but did not figure out. I just ended up running test in command line with bazel

I like the bazel-remote project, but storing the files on disk may not scale for our needs.

Some of the remote execution implementations (such as build-grid and build-barn) also allow storing the cache objects in S3.

I'd be nice for bazel-remote to have the same capability.

One of the uses of the remote cache is as a proxy on local machines (Bazel waits for all the artifacts to upload before completing the build so a slow network connection can impact build times).

I've been playing with the idea of having my Bazel wrapper automatically spawn a bazel-remote subprocess on a local port and rewire the Bazel command to prefer the local cache server than the remote one.

In order for this to work, there are two features that I think would be needed from bazel-remote:

1. Detect when a bazel-remote process is already serving objects from a directory.
2. Exit after an idle timeout.

I think for 1. the easiest would be to write the PID and port number in a file inside the cache directory. On startup, bazel-remote could check if that PID is still running, if not, it would start on the same port.

I could manage this from the Bazel wrapper, but I think it would be better to add it to bazel-remote so I'm opening an issue to start the discussion.

We have a case where some empty actions managed to sneak into our ac cache.

$ find ac -size 0 | head
ac/0a/0a372b8f6f57e408155b2d77291599fa69c3ad44c32a370579bc3fd582f462e6
ac/1d/1d16b14f8365aa77a51617086c6a6675fd216480223ffa02a5cbf33322570f8d
ac/87/875ac9c5592f5f8c86dc05d3c61d18a9d7aac85c0930b7b36290ba9f46fa3367

These are problematic because Bazel will fail the build when the outputs are not created.

Would it make sense for bazel-remote to refuse to serve empty actions?

Such as current size, number of objects, etc.

Bazel remote caching has issues with files updating during build. To avoid them I would like to disable writing to remote cache for desktop machines and only allow the CI server to do the writing. There is a bazel flag build --remote_upload_local_results=false that can help me, but developers can still accidentally write to the remote cache. Hence it's best to have two instances of the cache over the same data directory and make one of them read-only to be used by developers and another one read-write for the CI server to cache the "master" branch.

I propose adding a -read-only flag that will allow starting a read-only cache instance that will discard PUT requests.

Attempting this on Raspberry Pi 3

Command: docker run -v /tmp/bazel-cache-data:/data -p 9090:80 buchgr/bazel-remote-cache

Result:

standard_init_linux.go:190: exec user process caused "exec format error"

This repository cannot build with Bazel incompatible flags.
At least --incompatible_no_transitive_loads is failing, probably more.

bazelisk --migrate can be useful to find the issues.

Could you fix the issues?
Thanks

Incompatible flag --incompatible_no_implicit_file_export will break Bazel Remote Cache once Bazel 1.2.1 is released.

Please see the following CI builds for more information:

• [:ubuntu: 16.04 (OpenJDK 8)](:ubuntu: 16.04 (OpenJDK 8))
• [:darwin: (OpenJDK 8)](:darwin: (OpenJDK 8))
• [:ubuntu: 18.04 (OpenJDK 11)](:ubuntu: 18.04 (OpenJDK 11))

Questions? Please file an issue in https://github.com/bazelbuild/continuous-integration

Important: Please do NOT modify the issue title since that might break our tools.

One of the most important features for any remote cache (IMO) is metrics. We want to know how healthy the cache is and be notified if something changes.

Adding support for Prometheus would give us that functionality.

I was running your amazing cache with a very small cache size and noticed that new files were no longer added to the cache after some time. After reading the code, I think there are a few bugs in EnsureSpacer:

d, err := os.Open(cache.Dir())
// ...
files, err := d.Readdir(-1)
// ...
for _, fileinfo := range files {
  name := fileinfo.Name()
  // ...
}

• It seems like we're leaking file descriptors here because d is never closed. I think I've seen some errors related to too many files open in log files, but I'm not 100% sure that it is related to this.
• files will always be a slice containing exactly ac and cas. I think what we want to do here is find the oldest files in the cache and delete them. This could be done with filepath.Walk, similar to what we do in LoadExistingFiles(), but since we know what files are stored in the cache, we could maybe find a more efficient solution.
• fileinfo.Name() returns the base name of a file, not the path (https://golang.org/pkg/os/#FileInfo). Unfortunately, there is no way to get the full path from FileInfo.

With bazel-remote running on localhost and an http_proxy configured to a remote bazel-remote, when an object is looked up remotely and not found, the local bazel-remote responds with a 500 Internal Server error.

This is the output from Bazel (0.22.0):

 $ bazel test //...
INFO: Using Bazel Remote listening on port 55381
INFO: Running Bazel version 0.22.0.4
INFO: Invocation ID: 243c2863-f4f3-4a27-b35c-8ee3362b4990
INFO: Analysed 40 targets (0 packages loaded, 0 targets configured).
INFO: Found 38 targets and 2 test targets...
WARNING: Error reading from the remote cache:
500 Internal Server Error
Not found

INFO: Elapsed time: 0.886s, Critical Path: 0.63s
INFO: 1 process: 1 darwin-sandbox.
INFO: Build completed successfully, 2 total actions

There are no obvious errors in the local bazel-remote log, but I used tcpdump to see the packets:

This is Bazel:

GET /ac/5a776703f6b95900eba8e7a1779df127f0c61a60610e9e1322c94d3bbe3bd1fd HTTP/1.1
host: 127.0.0.1:55381
connection: keep-alive
accept: */*

This is the local bazel-remote log:

2019/03/01 19:27:46  GET 404                 http://bazel-remote.example.com:8415/ac/5a776703f6b95900eba8e7a1779df127f0c61a60610e9e1322c94d3bbe3bd1fd
2019/03/01 19:27:46 GET /ac/5a776703f6b95900eba8e7a1779df127f0c61a60610e9e1322c94d3bbe3bd1fd: Not found

This is the response from the local bazel-remote:

HTTP/1.1 500 Internal Server Error
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Fri, 01 Mar 2019 19:27:46 GMT
Content-Length: 11

Not found

The local bazel-remote should reply with an HTTP code 404 Not Found instead of a 500 Internal Server Error.

In this line, it's necessary to read the response and close the body, like so:

@@ -68,6 +68,10 @@ func uploadFile(remote *http.Client, baseURL *url.URL, local cache.Cache, access
        if err != nil {
                return
        }
+       ioutil.ReadAll(rsp.Body)
+       rsp.Body.Close()
+
+
        logResponse(accessLogger, "PUT", rsp.StatusCode, url)
        return
 }

or bazel-remote will leak file descriptors when uploading to a remote cache.

From the Go documentation:

// If the returned error is nil, the Response will contain a non-nil
// Body which the user is expected to close. If the Body is not both
// read to EOF and closed, the Client's underlying RoundTripper
// (typically Transport) may not be able to re-use a persistent TCP
// connection to the server for a subsequent "keep-alive" request.
//
// The request Body, if non-nil, will be closed by the underlying
// Transport, even on errors.

I am not convinced that we the remote cache should verify the hash sum of CAS entries, in that it costs significant processing power and it's not clear that it helps with anything :-). Wdyt @nicolov?

Probing http://mybazelremote.local/ returns error 400 causing Kubernetes liveness and readiness checks to fail. What path can we use to probe the server's health? eg. http://mybazelremote.local/healthz

Thanks.

I am using bazel-gazelle (0.18.2) and rules_go (0.19.4) to build bazel-remote. The GoSdk version is 1.13.

The workspace repository rule is

    go_repository(
        name = 'com_github_buchgr_bazel_remote',
        sha256 = r'''43de48c8c52078b98befe80ac51e8641f32e70c553fa91d9e4b174ee30c37788''',
        strip_prefix = r'''bazel-remote''',
        urls = ['http://git.corp.logiocean.com:3000/buchgr/bazel-remote/archive/c98f161a777d60b3f5dfa5748c904be764892594.tar.gz'],
        importpath = r'''github.com/buchgr/bazel-remote''',
    )

Then I ran

bazel build @com_github_buchgr_bazel_remote//:bazel-remote

I expect this to build correctly. But it gave me the following errors:

12:13 $ bazel build @com_github_buchgr_bazel_remote//:bazel-remote
INFO: Invocation ID: d805e18b-34b2-4e79-a601-3bd51d15ca75
INFO: Analyzed target @com_github_buchgr_bazel_remote//:bazel-remote (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
INFO: Writing explanation of rebuilds to 'build/logs'
ERROR: /home/zhongming/.cache/bazel/_bazel_zhongming/5a5efdfcd4f98edbd655a37c61128c16/external/com_github_buchgr_bazel_remote/BUILD.bazel:28:1: GoLink external/com_github_buchgr_bazel_remote/linux_amd64_static_pure_stripped/bazel-remote failed (Exit 1) builder failed: error executing command
  (cd /home/zhongming/.cache/bazel/_bazel_zhongming/5a5efdfcd4f98edbd655a37c61128c16/sandbox/linux-sandbox/32199/execroot/logi && \
  exec env - \
    CGO_ENABLED=0 \
    GOARCH=amd64 \
    GOOS=linux \
    GOROOT=bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_static_pure_stripped/stdlib% \
    GOROOT_FINAL=GOROOT \
    PATH=tools/toolchains/cc/clang/wrappers:/bin:/usr/bin \
  bazel-out/host/bin/external/go_sdk/builder link -sdk external/go_sdk -installsuffix linux_amd64 -arc '@com_github_buchgr_bazel_remote//cache:go_default_library=github.com/buchgr/bazel-remote/cache=bazel-out/k8-fastbuild/bin/external/com_github_buchgr_bazel_remote/cache/linux_amd64_static_pure_stripped/go_default_library%/github.com/buchgr/bazel-remote/cache.a' -arc '@com_github_buchgr_bazel_remote//cache/disk:go_default_library=github.com/buchgr/bazel-remote/cache/disk=bazel-out/k8-fastbuild/bin/external/com_github_buchgr_bazel_remote/cache/disk/linux_amd64_static_pure_stripped/go_default_library%/github.com/buchgr/bazel-remote/cache/disk.a' -arc '@com_github_djherbis_atime//:go_default_library=github.com/djherbis/atime=bazel-out/k8-fastbuild/bin/external/com_github_djherbis_atime/linux_amd64_static_pure_stripped/go_default_library%/github.com/djherbis/atime.a' -arc '@com_github_buchgr_bazel_remote//cache/gcs:go_default_library=github.com/buchgr/bazel-remote/cache/gcs=bazel-out/k8-fastbuild/bin/external/com_github_buchgr_bazel_remote/cache/gcs/linux_amd64_static_pure_stripped/go_default_library%/github.com/buchgr/bazel-remote/cache/gcs.a' -arc '@com_github_buchgr_bazel_remote//cache/http:go_default_library=github.com/buchgr/bazel-remote/cache/http=bazel-out/k8-fastbuild/bin/external/com_github_buchgr_bazel_remote/cache/http/linux_amd64_static_pure_stripped/go_default_library%/github.com/buchgr/bazel-remote/cache/http.a' -arc '@org_golang_x_oauth2//:go_default_library=golang.org/x/oauth2=bazel-out/k8-fastbuild/bin/external/org_golang_x_oauth2/linux_amd64_static_pure_stripped/go_default_library%/golang.org/x/oauth2.a' -arc '@org_golang_x_oauth2//internal:go_default_library=golang.org/x/oauth2/internal=bazel-out/k8-fastbuild/bin/external/org_golang_x_oauth2/internal/linux_amd64_static_pure_stripped/go_default_library%/golang.org/x/oauth2/internal.a' -arc '@org_golang_x_net//context/ctxhttp:go_default_library=golang.org/x/net/context/ctxhttp=bazel-out/k8-fastbuild/bin/external/org_golang_x_net/context/ctxhttp/linux_amd64_static_pure_stripped/go_default_library%/golang.org/x/net/context/ctxhttp.a' -arc '@org_golang_x_oauth2//google:go_default_library=golang.org/x/oauth2/google=bazel-out/k8-fastbuild/bin/external/org_golang_x_oauth2/google/linux_amd64_static_pure_stripped/go_default_library%/golang.org/x/oauth2/google.a' -arc '@org_golang_x_oauth2//jws:go_default_library=golang.org/x/oauth2/jws=bazel-out/k8-fastbuild/bin/external/org_golang_x_oauth2/jws/linux_amd64_static_pure_stripped/go_default_library%/golang.org/x/oauth2/jws.a' -arc '@org_golang_x_oauth2//jwt:go_default_library=golang.org/x/oauth2/jwt=bazel-out/k8-fastbuild/bin/external/org_golang_x_oauth2/jwt/linux_amd64_static_pure_stripped/go_default_library%/golang.org/x/oauth2/jwt.a' -arc '@com_google_cloud_go//compute/metadata:go_default_library=cloud.google.com/go/compute/metadata=bazel-out/k8-fastbuild/bin/external/com_google_cloud_go/compute/metadata/linux_amd64_static_pure_stripped/go_default_library%/cloud.google.com/go/compute/metadata.a' -arc '@com_github_buchgr_bazel_remote//config:go_default_library=github.com/buchgr/bazel-remote/config=bazel-out/k8-fastbuild/bin/external/com_github_buchgr_bazel_remote/config/linux_amd64_static_pure_stripped/go_default_library%/github.com/buchgr/bazel-remote/config.a' -arc '@in_gopkg_yaml_v2//:go_default_library=gopkg.in/yaml.v2=bazel-out/k8-fastbuild/bin/external/in_gopkg_yaml_v2/linux_amd64_static_pure_stripped/go_default_library%/gopkg.in/yaml.v2.a' -arc '@com_github_buchgr_bazel_remote//server:go_default_library=github.com/buchgr/bazel-remote/server=bazel-out/k8-fastbuild/bin/external/com_github_buchgr_bazel_remote/server/linux_amd64_static_pure_stripped/go_default_library%/github.com/buchgr/bazel-remote/server.a' -arc '@com_github_abbot_go_http_auth//:go_default_library=github.com/abbot/go-http-auth=bazel-out/k8-fastbuild/bin/external/com_github_abbot_go_http_auth/linux_amd64_static_pure_stripped/go_default_library%/github.com/abbot/go-http-auth.a' -arc '@org_golang_x_crypto//bcrypt:go_default_library=golang.org/x/crypto/bcrypt=bazel-out/k8-fastbuild/bin/external/org_golang_x_crypto/bcrypt/linux_amd64_static_pure_stripped/go_default_library%/golang.org/x/crypto/bcrypt.a' -arc '@org_golang_x_crypto//blowfish:go_default_library=golang.org/x/crypto/blowfish=bazel-out/k8-fastbuild/bin/external/org_golang_x_crypto/blowfish/linux_amd64_static_pure_stripped/go_default_library%/golang.org/x/crypto/blowfish.a' -arc '@org_golang_x_net//context:go_default_library=golang.org/x/net/context=bazel-out/k8-fastbuild/bin/external/org_golang_x_net/context/linux_amd64_static_pure_stripped/go_default_library%/golang.org/x/net/context.a' -arc '@com_github_urfave_cli//:go_default_library=github.com/urfave/cli=bazel-out/k8-fastbuild/bin/external/com_github_urfave_cli/linux_amd64_static_pure_stripped/go_default_library%/github.com/urfave/cli.a' -package_list bazel-out/host/bin/external/go_sdk/packages.txt -X 'github.com/buchgr/bazel-remote/server.GitCommit={STABLE_GIT_COMMIT}' -o bazel-out/k8-fastbuild/bin/external/com_github_buchgr_bazel_remote/linux_amd64_static_pure_stripped/bazel-remote -main bazel-out/k8-fastbuild/bin/external/com_github_buchgr_bazel_remote/linux_amd64_static_pure_stripped/bazel-remote%/github.com/buchgr/bazel-remote.a -p github.com/buchgr/bazel-remote -- -extld tools/toolchains/cc/clang/wrappers/clang -linkmode external '-buildid=redacted' -w -extldflags '-Lexternal/clang/lib -lc++abi -lm -fuse-ld=lld -Wl,-z,relro,-z,now -no-canonical-prefixes -static')
Execution platform: @bazel_tools//platforms:host_platform

Use --sandbox_debug to see verbose messages from the sandbox
external/go_sdk/pkg/tool/linux_amd64/link: running tools/toolchains/cc/clang/wrappers/clang failed: exit status 1
ld.lld: error: undefined symbol: x_cgo_callers
>>> referenced by go.go
>>>               /tmp/go-link-481443378/go.o:(_cgo_callers)

ld.lld: error: undefined symbol: x_cgo_init
>>> referenced by go.go
>>>               /tmp/go-link-481443378/go.o:(_cgo_init)

ld.lld: error: undefined symbol: x_cgo_mmap
>>> referenced by go.go
>>>               /tmp/go-link-481443378/go.o:(_cgo_mmap)

ld.lld: error: undefined symbol: x_cgo_munmap
>>> referenced by go.go
>>>               /tmp/go-link-481443378/go.o:(_cgo_munmap)

ld.lld: error: undefined symbol: x_cgo_notify_runtime_init_done
>>> referenced by go.go
>>>               /tmp/go-link-481443378/go.o:(_cgo_notify_runtime_init_done)

ld.lld: error: undefined symbol: x_cgo_sigaction
>>> referenced by go.go
>>>               /tmp/go-link-481443378/go.o:(_cgo_sigaction)

ld.lld: error: undefined symbol: x_cgo_thread_start
>>> referenced by go.go
>>>               /tmp/go-link-481443378/go.o:(_cgo_thread_start)

ld.lld: error: undefined symbol: x_cgo_setenv
>>> referenced by go.go
>>>               /tmp/go-link-481443378/go.o:(runtime._cgo_setenv)

ld.lld: error: undefined symbol: x_cgo_unsetenv
>>> referenced by go.go
>>>               /tmp/go-link-481443378/go.o:(runtime._cgo_unsetenv)

ld.lld: error: undefined symbol: _cgo_yield
>>> referenced by go.go
>>>               /tmp/go-link-481443378/go.o:(runtime.cgo_yield)
clang: error: linker command failed with exit code 1 (use -v to see invocation)

link: error running subcommand: exit status 2
Target @com_github_buchgr_bazel_remote//:bazel-remote failed to build
INFO: Elapsed time: 0.564s, Critical Path: 0.38s
INFO: 0 processes.
FAILED: Build did NOT complete successfully

I have been building this binary using GoSdk 1.11 through 1.12. Those versions (with their compatible versions of rules_go and bazel-gazelle) of GoSdk built bazel-remote correctly.

I know very little about cgo. But this error looks strange to me as I did not expect bazel-remote to have any cgo code.

Help or pointers are appreciated.

Thanks!

@BenTheElder mentioned to me that k8s is using a different remote cache for different projects and they distinguish between projects via a path component in the URL:

https://cache.com/<project name>/(ac|cas)/

I think that's a good idea and I believe @nicolov already laid the foundations for that.

I think it would be helpful if the README.md described the REST API, and the difference between the action cache and the CAS cache.

After poking around a bit I found some info in #5, if you like I can have a go at documenting this?

We had a case of cache poisoning where to fix, we attempted to delete the bad file, this resulted in this server error:

WARNING: Reading from Remote Cache:
500 Internal Server Error
stat /data/cas/a25641977eb4f12d165dda1a1459c1c8c449ffb1146b90cfec5176c1ca85f3db: no such file or directory

It would be nice to handle this gracefully and 404

On startup, bazel-remote scans the cache directory and sorts cache files by their atime. The Get and Put functions update the in-memory LRU index and open the file which updates the atime, but the Contains function only updates the in-memory LRU index. When bazel-remote restarts, the in-memory LRU index is lost and the new index that is recalculated will miss the effects of Contains.

Should bazel-remote update the atimes of cache files when Contains is called? This would give more correct results after restarts, but might be slower. Or should we implement a graceful shutdown process that flushes the in-memory index to disk (either by touching all the cache files in LRU order, or by explicitly dumping the in-memory LRU to disk and using that at startup time)?

I think the server should also:

• accept gzip-compressed PUT
• stream gzip-compressed PUT into a $hash.gz file without decompressing (except in a parallel stream just to check the incoming hash)
• serve a gzip-compressed stream to the bazel client from a .gz file without decompressing.

Object files compress very well, so this will save significant amount of IO/bandwidth. What do you think @buchgr?

Related to #59, I have a setup where developer machines have a local cache they write to, as discussed in bazelbuild/bazel#5139, however to proxy to the shared cache that CI writes to one must also enable writing to it, as there is no configuration knob for that. I could disable writing to the cache on the client side, but then we wouldn't be using the cache running locally as a directory cache.

I was thinking of creating an Http middleware to do some logging. @buchgr did you have other ideas in mind?

I feel like caching metadata in memory would be super efficient. Is it worth implementing? @buchgr

I can't seem to build this project. I get the following error when running bazel build ...:

compilepkg: missing strict dependencies: ... import of "golang.org/x/sys/unix"
No dependencies were provided.
Check that imports in Go sources match importpath attributes in deps.

I'm able to pull down the latest code, build it, and run it (admittedly without exhaustive testing), but was curious if anyone would be pushing the latest to DockerHub in the near future.

I'm seeing errors like:

process bazel-remote-0.0[61853] caught waking the CPU 45001 times over ~42 seconds, averaging 1066 wakes / second and violating a limit of 45000 wakes over 300 seconds.

my configuration uses idle_timeout so I suspect we are hitting a Go runtime bug: golang/go#27707

Still investigating but wanted to file the issue in case anyone hits it or has any ideas.

It would be great if we could add a way to use this cache as a library (i.e. to make integration into existing projects easier). As a side effect, this would mostly implement #12, except the configuration part, and make it easier for folks to implement their own storage mechanism on top of GCS, S3, ...

We would create a new package httpcache (to avoid conflicts with net/http), which has a function New that returns a http.Handler.

type Options struct {
  // Logging, ...
}

// New creates a new Bazel cache which uses the provided StorageMechanism
// to store and retrieve cached files.
func New(StorageMechanism, Options) http.Handler {
  // ...
}

(I think StorageMechanism will look mostly like LRU @nicolov implemented recently).

WDYT?