Comments (7)
It looks like a referenced CAS entry was expired - I would try bumping up your CAS byte count in your server's configuration as a test.
As to why this requires a server shutdown, I'm unsure: can you identify the rev of buildfarm running on both worker and server?
I have some pending changes to make the worker survive something like this a little more gracefully - in that case the work will be requeued for server validation. I will prioritize getting those changes into bazelbuild/master regardless.
from bazel-buildfarm.
As to why this requires a server shutdown, I'm unsure: can you identify the rev of buildfarm running on both worker and server?
(these are images built off #128)
from bazel-buildfarm.
I've set the CAS max size to LONG.MAX_VALUE, so I don't think that's it.
The worker ends up requesting a zero-length blob from the server that is NOT_FOUND. That blob hash does not seem to be populated in the contentaddressablestorage, so I think the reply from the server is correct. Still looking into it.
read request resource_name: "default_memory_instance/blobs/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/0"
readBlob default_memory_instance/blobs/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/0 using instance build.buildfarm.instance.memory.MemoryInstance@7fa1cd21
getBlob(hash: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
, 0, 0) --> NULL build.buildfarm.instance.memory.MemoryLRUContentAddressableStorage@53941ba3
from bazel-buildfarm.
ah hah! you need a recent change that prevents empty blob fetches. Note the /0
at the end of that digest name
from bazel-buildfarm.
Was that merged already? Again I'm currently off #128 but can cherry-pick that.
from bazel-buildfarm.
I think that's a good change, but I applied it directly and has not solved the problem (does not suppress that request to the server). Here's the file it's trying to link:
linking /tmp/worker/default_memory_instance/operations/d2b5da3b-dc85-4283-983b-0ae14aefd31e/bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_stripped/stdlib~/ROOT -> hash: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
That file actually exists (non-remote build) and is empty:
du -h bazel-bin/external/io_bazel_rules_go/linux_amd64_stripped/stdlib~/ROOT
0 bazel-bin/external/io_bazel_rules_go/linux_amd64_stripped/stdlib~/ROOT
Perhaps a change in CASFileCache.put to skip the request for zero-length blob? (just create a symlink right away, skip the tmpPath step)
I have to shift gears here soon but will continue later in the day.
from bazel-buildfarm.
similar to #150, can you provide a remote.log of the action in question? it will make incremental debugging of this issue easier.
from bazel-buildfarm.
Related Issues (20)
- Buildfarm unit tests are failing in windows at Bazel@HEAD in Downstream CI HOT 1
- Runtime metrics are not reported for match/report result stages
- helm chart: `ingress:` in values.yaml does nothing
- Upgrade jedis to v5, abandon fork
- Redis password is printed to the logs HOT 1
- Blocked thread in RedisShardBackplane HOT 5
- RedisPriorityQueue is malformed
- prometheus metric is malformed HOT 1
- `cas: false` not being respected? HOT 1
- Update to PMD 7.0.0
- How does access control work? HOT 7
- parse CA file with --tls_certificate in bazel does not work HOT 2
- log messages `WARNING: write: uploads/ ... no available workers`. HOT 4
- Work directory should be added automatically as an input directory HOT 3
- Many `WARNING: DEADLINE_EXCEEDED: write` logs from buildfarm-server. HOT 2
- [Bazel CI] gaussian_distribution_test is failing at Bazel@HEAD HOT 1
- No logs HOT 4
- Running a server on Linux and build on MacOS HOT 3
- buildfarm-worker on Windows Server 2022 fails to clean up operation files HOT 3
- if redisUri is empty, java.lang.NullPointerException error occurred HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bazel-buildfarm.