Comments (19)
tl;dr: no, dirs != files, but output_directories seem like the way to go
Ah, that's definitely a no-go.
There is, however, this:
However its implementation in bazel 0.9 looks like this:
And I was very excited to find when I went looking for this, that output_directories now has an implementation in the latest unreleased bazel:
For which I will happily mirror an implementation in buildfarm. My guess on documentation for how to add an entry to the output_directories
list is that it will also be supplied in 0.10. Thanks @ulfjack!
from bazel-buildfarm.
If I use "declary_directory" instead of "declare_file" I receive following problem with symlink to system python (only under linux, under OSX everything works fine) (just local bazel, not farm)
Is it expected behavior or another bug in bazel?
ERROR: /home/jenkins/backend-ci/plato/BUILD.bazel:45:1: Error while validating output TreeArtifact File:[[/home/jenkins/.cache/bazel/_bazel_jenkins/cff2319697c0ff2c301458e924e0bef7/execroot/XXX]bazel-out/k8-dbg/bin]plato/tests.virtualenv : /home/jenkins/.cache/bazel/_bazel_jenkins/cff2319697c0ff2c301458e924e0bef7/execroot/XXX/bazel-out/k8-dbg/bin/plato/tests.virtualenv/lib/python3.6/config-3.6m-x86_64-linux-gnu (Operation not permitted)
ERROR: /home/jenkins/backend-ci/plato/BUILD.bazel:45:1: not all outputs were created or valid
[jenkins@ci-slave-1:Ireland backend-ci]$ ls -al /home/jenkins/.cache/bazel/_bazel_jenkins/cff2319697c0ff2c301458e924e0bef7/execroot/XXX/bazel-out/k8-dbg/bin/plato/tests.virtualenv/lib/python3.6/
total 44
dr-xr-xr-x. 5 jenkins jenkins 4096 Jan 12 19:27 .
dr-xr-xr-x. 3 jenkins jenkins 23 Jan 12 19:27 ..
lrwxrwxrwx. 1 jenkins jenkins 27 Jan 12 19:27 abc.py -> /usr/lib64/python3.6/abc.py
lrwxrwxrwx. 1 jenkins jenkins 30 Jan 12 19:27 base64.py -> /usr/lib64/python3.6/base64.py
lrwxrwxrwx. 1 jenkins jenkins 30 Jan 12 19:27 bisect.py -> /usr/lib64/python3.6/bisect.py
lrwxrwxrwx. 1 jenkins jenkins 35 Jan 12 19:27 _bootlocale.py -> /usr/lib64/python3.6/_bootlocale.py
lrwxrwxrwx. 1 jenkins jenkins 30 Jan 12 19:27 codecs.py -> /usr/lib64/python3.6/codecs.py
lrwxrwxrwx. 1 jenkins jenkins 32 Jan 12 19:27 collections -> /usr/lib64/python3.6/collections
lrwxrwxrwx. 1 jenkins jenkins 40 Jan 12 19:27 _collections_abc.py -> /usr/lib64/python3.6/_collections_abc.py
lrwxrwxrwx. 1 jenkins jenkins 49 Jan 12 19:27 config-3.6m-x86_64-linux-gnu -> /usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu
lrwxrwxrwx. 1 jenkins jenkins 28 Jan 12 19:27 copy.py -> /usr/lib64/python3.6/copy.py
lrwxrwxrwx. 1 jenkins jenkins 31 Jan 12 19:27 copyreg.py -> /usr/lib64/python3.6/copyreg.py
drwxr-xr-x. 3 jenkins jenkins 65 Jan 12 19:27 distutils
lrwxrwxrwx. 1 jenkins jenkins 37 Jan 12 19:27 _dummy_thread.py -> /usr/lib64/python3.6/_dummy_thread.py
lrwxrwxrwx. 1 jenkins jenkins 30 Jan 12 19:27 encodings -> /usr/lib64/python3.6/encodings
lrwxrwxrwx. 1 jenkins jenkins 28 Jan 12 19:27 enum.py -> /usr/lib64/python3.6/enum.py
lrwxrwxrwx. 1 jenkins jenkins 31 Jan 12 19:27 fnmatch.py -> /usr/lib64/python3.6/fnmatch.py
lrwxrwxrwx. 1 jenkins jenkins 33 Jan 12 19:27 functools.py -> /usr/lib64/python3.6/functools.py
lrwxrwxrwx. 1 jenkins jenkins 34 Jan 12 19:27 __future__.py -> /usr/lib64/python3.6/__future__.py
lrwxrwxrwx. 1 jenkins jenkins 35 Jan 12 19:27 genericpath.py -> /usr/lib64/python3.6/genericpath.py
lrwxrwxrwx. 1 jenkins jenkins 31 Jan 12 19:27 hashlib.py -> /usr/lib64/python3.6/hashlib.py
lrwxrwxrwx. 1 jenkins jenkins 29 Jan 12 19:27 heapq.py -> /usr/lib64/python3.6/heapq.py
lrwxrwxrwx. 1 jenkins jenkins 28 Jan 12 19:27 hmac.py -> /usr/lib64/python3.6/hmac.py
lrwxrwxrwx. 1 jenkins jenkins 30 Jan 12 19:27 importlib -> /usr/lib64/python3.6/importlib
lrwxrwxrwx. 1 jenkins jenkins 27 Jan 12 19:27 imp.py -> /usr/lib64/python3.6/imp.py
lrwxrwxrwx. 1 jenkins jenkins 26 Jan 12 19:27 io.py -> /usr/lib64/python3.6/io.py
lrwxrwxrwx. 1 jenkins jenkins 31 Jan 12 19:27 keyword.py -> /usr/lib64/python3.6/keyword.py
lrwxrwxrwx. 1 jenkins jenkins 32 Jan 12 19:27 lib-dynload -> /usr/lib64/python3.6/lib-dynload
lrwxrwxrwx. 1 jenkins jenkins 33 Jan 12 19:27 linecache.py -> /usr/lib64/python3.6/linecache.py
lrwxrwxrwx. 1 jenkins jenkins 30 Jan 12 19:27 locale.py -> /usr/lib64/python3.6/locale.py
-rw-r--r--. 1 jenkins jenkins 0 Jan 12 19:27 no-global-site-packages.txt
lrwxrwxrwx. 1 jenkins jenkins 30 Jan 12 19:27 ntpath.py -> /usr/lib64/python3.6/ntpath.py
lrwxrwxrwx. 1 jenkins jenkins 32 Jan 12 19:27 operator.py -> /usr/lib64/python3.6/operator.py
-rw-r--r--. 1 jenkins jenkins 4 Jan 12 19:27 orig-prefix.txt
lrwxrwxrwx. 1 jenkins jenkins 26 Jan 12 19:27 os.py -> /usr/lib64/python3.6/os.py
lrwxrwxrwx. 1 jenkins jenkins 33 Jan 12 19:27 posixpath.py -> /usr/lib64/python3.6/posixpath.py
drwxr-xr-x. 2 jenkins jenkins 4096 Jan 12 19:27 __pycache__
lrwxrwxrwx. 1 jenkins jenkins 30 Jan 12 19:27 random.py -> /usr/lib64/python3.6/random.py
lrwxrwxrwx. 1 jenkins jenkins 31 Jan 12 19:27 reprlib.py -> /usr/lib64/python3.6/reprlib.py
lrwxrwxrwx. 1 jenkins jenkins 26 Jan 12 19:27 re.py -> /usr/lib64/python3.6/re.py
lrwxrwxrwx. 1 jenkins jenkins 35 Jan 12 19:27 rlcompleter.py -> /usr/lib64/python3.6/rlcompleter.py
lrwxrwxrwx. 1 jenkins jenkins 30 Jan 12 19:27 shutil.py -> /usr/lib64/python3.6/shutil.py
drwxr-xr-x. 66 jenkins jenkins 4096 Jan 12 19:27 site-packages
-rw-r--r--. 1 jenkins jenkins 27543 Jan 12 19:27 site.py
lrwxrwxrwx. 1 jenkins jenkins 35 Jan 12 19:27 sre_compile.py -> /usr/lib64/python3.6/sre_compile.py
lrwxrwxrwx. 1 jenkins jenkins 37 Jan 12 19:27 sre_constants.py -> /usr/lib64/python3.6/sre_constants.py
lrwxrwxrwx. 1 jenkins jenkins 33 Jan 12 19:27 sre_parse.py -> /usr/lib64/python3.6/sre_parse.py
lrwxrwxrwx. 1 jenkins jenkins 28 Jan 12 19:27 stat.py -> /usr/lib64/python3.6/stat.py
lrwxrwxrwx. 1 jenkins jenkins 30 Jan 12 19:27 struct.py -> /usr/lib64/python3.6/struct.py
lrwxrwxrwx. 1 jenkins jenkins 31 Jan 12 19:27 tarfile.py -> /usr/lib64/python3.6/tarfile.py
lrwxrwxrwx. 1 jenkins jenkins 32 Jan 12 19:27 tempfile.py -> /usr/lib64/python3.6/tempfile.py
lrwxrwxrwx. 1 jenkins jenkins 32 Jan 12 19:27 tokenize.py -> /usr/lib64/python3.6/tokenize.py
lrwxrwxrwx. 1 jenkins jenkins 29 Jan 12 19:27 token.py -> /usr/lib64/python3.6/token.py
lrwxrwxrwx. 1 jenkins jenkins 29 Jan 12 19:27 types.py -> /usr/lib64/python3.6/types.py
lrwxrwxrwx. 1 jenkins jenkins 32 Jan 12 19:27 warnings.py -> /usr/lib64/python3.6/warnings.py
lrwxrwxrwx. 1 jenkins jenkins 31 Jan 12 19:27 weakref.py -> /usr/lib64/python3.6/weakref.py
lrwxrwxrwx. 1 jenkins jenkins 35 Jan 12 19:27 _weakrefset.py -> /usr/lib64/python3.6/_weakrefset.py
from bazel-buildfarm.
I'm going to have to punt to the bazel project for non-buildfarm behaviors around declare_directory
from bazel-buildfarm.
@werkt Yes. My latest comment about local bazel (not a farm).
Can you please give me advise - is it expected behavior or bug to bazel which I should report to bazelbuild/bazel?
from bazel-buildfarm.
Looks like a bazel bug to me.
My clue: Error while validating output TreeArtifact
- I usually don't expect to see classname mentions in expected behavior.
from bazel-buildfarm.
For which I will happily mirror an implementation in buildfarm.
@werkt I am very sorry for probably stupid question...
Why you do not want to use bazel as external repo in farm?
As result you can import libraries from bazel directly to farm and 99% to have correct behavior always
from bazel-buildfarm.
Not stupid, I would say that there are two reasons:
- my implementation of this (or almost any other mechanism) will probably be dissimilar, and would need to accommodate a reflection that will not be quite the same during the workers evaluation to anything implemented in bazel; my guidance for how it should behave had no model in an existing implementation for download, which is why I needed some treatment of it from bazel's side to get started (and I keep forgetting to ask the bazel/remote ex team what their intention was)
- the other is that using bazel sources directly without filtering their packaging requires that I pull in dependencies in the groupings that they've defined for them - the remote ex interactions that I'm interested in are not isolated completely in bazel sources. I'd be reliant or committed to maintaining boundaries in their code, and tbh I don't feel we need to be beholden to it. We already share a common endpoint in googleapis, and that has been helpful enough for me to remain consistent with them.
from bazel-buildfarm.
Thank you!
Wait for fix for double check
from bazel-buildfarm.
Any updates?
from bazel-buildfarm.
No movement. Happy to accept a PR if you want this to be accelerated.
from bazel-buildfarm.
You may be interested in bazelbuild/bazel#4668, which I encountered when trying to demonstrate a solution for this. I'll have to proceed without an integration test on this issue until they fix that.
from bazel-buildfarm.
@werkt That's exactly my problem.
That's why my virtualenv rule (and rules_go) use "declare_file" and store directory instead of file in result.
If you replace "declare_directory" by "declare_file" everything would works fine.
from bazel-buildfarm.
from bazel-buildfarm.
@werkt thank you very much for update!
from bazel-buildfarm.
https://github.com/werkt/bazel-buildfarm/tree/output-directories provides an implementation for buildfarm, according to the observed behavior in bazel, and includes some tests. Don't know what to say for in-bazel support for legit output directory specification, but at least we'll be ready...
from bazel-buildfarm.
@werkt good news! I will check in nearest future how buildfarm works now
from bazel-buildfarm.
@werkt I tested https://github.com/werkt/bazel-buildfarm/tree/output-directories and it works.
from bazel-buildfarm.
Thanks @pauldraper, the PR is only currently held up by me - there is substantially strange behavior that needs to be considered when working with output directories, specifically intersections with other output files/dirs, and other inputs. The initial behavior will be to reject these outliers, and I have to clean things up a little more in the change to reject those conditions as invalid initially.
from bazel-buildfarm.
Implemented in #128 and landed
from bazel-buildfarm.
Related Issues (20)
- Do deployments via Helm work as RBE ? HOT 5
- make execution-policy=as-nobody the default on Linux
- [ZstdCompression] Read request with offset incorrectly fails with not_found error and removes blob
- [Bazel CI] apt-get and integration tests are failing with Bazel@HEAD HOT 1
- Tips on learning buildfarm? HOT 2
- Remote builds stuck on Buildfarm that is deployed with Helm. HOT 6
- Implement Fetch 'Push' Service
- Support http_header: prefix in Fetch Qualifiers
- bazel.canonical_id unsupported in Asset Fetch API HOT 1
- ERROR: error running 'git fetch origin refs/heads/*:refs/remotes/origin/* refs/tags/*:refs/tags/*' while working with @build_buildfarm~: HOT 2
- Updated from java17 to java21 to use the new server and worker causes JVM to shutdown
- Fetch service does not properly handle missing content-length HOT 1
- Support Multiple Hashing Function
- Remote execution service executing processes locally HOT 4
- ERROR: Failed to query remote execution capabilities: UNAVAILABLE: io exception HOT 4
- Feasibility Analysis of Using Buildfarm for Large-Scale Development HOT 2
- Verbose Logging for Servers and Workers through Helm Chart
- Check logs during remote execution HOT 4
- Incorrect container port for Shard-worker in Helm template
- hardlinks in CAS leads to task failure in some cases HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bazel-buildfarm.