Comments (12)
https://github.com/cdlee2/tools_remote-client is the canonical tool for interacting with those log files
from bazel-buildfarm.
Actually, I can see the problem from here: if you're operating with the default CAS size from examples/server.config.example
, then you don't have enough storage to hold all of the inputs for your actions: your action inputs sum to over 1GiB for the action you posted. Buildfarm is evicting your existing content in order to store each of your missing blobs, and pushing out further needed content.
Your CAS must always be at least as big as your largest action, or it will never build successfully:
# limit for CAS total content size in bytes
cas_max_size_bytes: 1073741824 # 1024 * 1024 * 1024
I recommend setting that a bit higher
from bazel-buildfarm.
Much though I would like to debug your client's remote execution implementation, I am not responsible for this google.devtools.build
of which you speak. I might recommend outputting some logging, perhaps with a --experimental_remote_grpc_log
and attaching it here
from bazel-buildfarm.
$ bazel build --spawn_strategy=remote --genrule_strategy=remote --strategy=Javac=remote --strategy=Closure=remote --remote_executor=server:8980 --experimental_remote_grpc_log media_service
reports success. Retrying with --experimental_remote_grpc_log=remote.log
fails as expected.
This log won't contain any source, will it? I don't see any when opening with a text editor, but I want to make sure before attaching it 😃
from bazel-buildfarm.
I threw together a trivial LogEntry protobuf decoder, but it complains of invalid utf-8 - so it's safe to say that I'm doing something wrong.
from bazel-buildfarm.
There should be no source included in those logs, but there is the capacity for ActionResults to contain output artifacts, and I don't want to recommend that you expose those if you're concerned.
I've put up a branch of buildfarm at https://github.com/werkt/bazel-buildfarm/tree/missing-input-logging that will include the digests of the files/directories that are missing when execute is failing in that way. You can then use the remote_client to correlate the requests over those missing digests, and identify the activities performed and reduce the logging attachment required to present the issue.
from bazel-buildfarm.
remote.log
The log looked clean, so I uploaded it.
FWIW, I encountered this error with bazel 0.14.0 and with a version that includes an s3 patch. I'm not sure exactly what official version that most closely resembles though...
from bazel-buildfarm.
Give this a try with the missing-input-logging tree, as it will help identify what is going wrong with your actions.
from bazel-buildfarm.
Will do
from bazel-buildfarm.
Going to close this out, as that has to have been the issue, and buildfarm was responding somewhat appropriately (not crashing at least) - feel free to reopen if you have a feature request with requirements in mind.
from bazel-buildfarm.
Looks like you solved this minutes before I finally got around to rebuilding with your missing-input-logging branch 😀
If I understand correctly, the related changes are PRs #155, #156 which are merged. I'll edit the CAS size limit, rebuild, and report success/failure.
from bazel-buildfarm.
Success - thanks!
from bazel-buildfarm.
Related Issues (20)
- Do deployments via Helm work as RBE ? HOT 5
- make execution-policy=as-nobody the default on Linux
- [ZstdCompression] Read request with offset incorrectly fails with not_found error and removes blob
- [Bazel CI] apt-get and integration tests are failing with Bazel@HEAD HOT 1
- Tips on learning buildfarm? HOT 2
- Remote builds stuck on Buildfarm that is deployed with Helm. HOT 6
- Implement Fetch 'Push' Service
- Support http_header: prefix in Fetch Qualifiers
- bazel.canonical_id unsupported in Asset Fetch API HOT 1
- ERROR: error running 'git fetch origin refs/heads/*:refs/remotes/origin/* refs/tags/*:refs/tags/*' while working with @build_buildfarm~: HOT 2
- Updated from java17 to java21 to use the new server and worker causes JVM to shutdown
- Fetch service does not properly handle missing content-length HOT 1
- Support Multiple Hashing Function
- Remote execution service executing processes locally HOT 4
- ERROR: Failed to query remote execution capabilities: UNAVAILABLE: io exception HOT 4
- Feasibility Analysis of Using Buildfarm for Large-Scale Development HOT 2
- Verbose Logging for Servers and Workers through Helm Chart
- Check logs during remote execution HOT 4
- Incorrect container port for Shard-worker in Helm template
- hardlinks in CAS leads to task failure in some cases HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bazel-buildfarm.