Comments (28)
So I think we wouldn't actually ship a symlink on anything but Linux.
Any platform except Windows you mean?
As far as I can tell, dist-x86_64-linux is the only configuration where we link LLVM dynamically. (Well and the llvm-16/llvm-17 images, but that's non-dist and a different situation anyway.)
from rust.
How does miri link against rust?
from rust.
Just a bunch of extern crate
:
extern crate rustc_apfloat;
extern crate rustc_ast;
extern crate rustc_const_eval;
extern crate rustc_data_structures;
extern crate rustc_errors;
extern crate rustc_hir;
extern crate rustc_index;
#[macro_use]
extern crate rustc_middle;
extern crate rustc_session;
extern crate rustc_span;
extern crate rustc_target;
// Necessary to pull in object code as the rest of the rustc crates are shipped only as rmeta
// files.
#[allow(unused_extern_crates)]
extern crate rustc_driver;
from rust.
Given that rustc CI still works and only builds against the distributed rustc toolchain are affected, it seems likely that the issue is related to the rustc-dev component -- maybe that doesn't contain all the required files any more?
from rust.
Reproducing instructions:
- checkout out this Miri branch
./miri toolchain
to install the right toolchain (that branch uses eaee1e9)./miri build
from rust.
Given that rustc CI still works and only builds against the distributed rustc toolchain are affected, it seems likely that the issue is related to the rustc-dev component -- maybe that doesn't contain all the required files any more?
The necessary symlink is indeed not part of rustc-dev (only rust-dev). We could add it, but it's not obvious to me why it is needed. I'm not familiar with how rustc generates that linker invocation, but the fact that it explicitly includes -lLLVM
in it seems wrong to me (-lLLVM
is a dependency of -lrustc_driver
and just linking against -lrustc_driver
should be sufficient).
from rust.
The necessary symlink is indeed not part of rustc-dev (only rust-dev).
There is no rust-dev
component? At least this doesn't list one and rustup +nightly component list | grep rust-dev
comes back empty.
We could add it, but it's not obvious to me why it is needed. I'm not familiar with how rustc generates that linker invocation, but the fact that it explicitly includes -lLLVM in it seems wrong to me (-lLLVM is a dependency of -lrustc_driver and just linking against -lrustc_driver should be sufficient).
I don't know anything about this linking magic either.^^ Cc @bjorn3
from rust.
But evidently the symlink is needed... so in the mean time it'd be good to unstuck this by adding the symlink I think. I expect this will break out-of-tree builds of Clippy as well, and rustfmt -- basically every tool that links against rustc.
from rust.
The necessary symlink is indeed not part of rustc-dev (only rust-dev).
There is no
rust-dev
component? At least this doesn't list one andrustup +nightly component list | grep rust-dev
comes back empty.
Ah, it's not a real component, just the tarball that download-ci-llvm uses.
But evidently the symlink is needed... so in the mean time it'd be good to unstuck this by adding the symlink I think. I expect this will break out-of-tree builds of Clippy as well, and rustfmt -- basically every tool that links against rustc.
Main problem is that we have checks against shipping symlinks, and I expect they exist for good reason...
from rust.
Ah, yeah probably Windows doesn't like them.
Why can't rustc_driver link against the correct .so file, i.e., why is a symlink needed in the first place?
from rust.
I think this is the reason why it gets added to the linker line:
rust/compiler/rustc_codegen_ssa/src/back/link.rs
Lines 2708 to 2711 in 4cdd205
Why can't rustc_driver link against the correct .so file, i.e., why is a symlink needed in the first place?
This is because llvm-config returns something like -lLLVM-18
and the symlink is needed to resolve that to -l:libLLVM.so.18.1
. And it seems like rustc just embeds the original name in the dylib, rather than the symlink-resolved variant, so the symlink is still needed.
Possibly that's a bug? Similar to how shared objects embed the resolved name, maybe rustc should do the same for its own metadata.
Though independent of general rustc behavior, I guess we could explicitly resolve the symlink in the build.rs for rustc_llvm. That might be the most straightforward fix.
from rust.
This is because llvm-config returns something like -lLLVM-18 and the symlink is needed to resolve that to -l:libLLVM.so.18.1
If llvm-config inherently relies on a symlink, how do they support Windows?
from rust.
Possibly that's a bug? Similar to how shared objects embed the resolved name, maybe rustc should do the same for its own metadata.
That would break compiling an rlib which states that it links against a certain dylib when said dylib is not available on the host, right? Currently you only need dylibs to be available when actually linking (that is using a crate type other than rlib or staticlib).
I'm not familiar with how rustc generates that linker invocation, but the fact that it explicitly includes -lLLVM in it seems wrong to me (-lLLVM is a dependency of -lrustc_driver and just linking against -lrustc_driver should be sufficient).
It includes -lLLVM-18-rust-1.78.0-nightly
, not -lLLVM
. Even if just -lrustc_driver
was passed, librustc_driver.so
has a DT_NEEDED
for LLVM-18-rust-1.78.0-nightly
and thus libLLVM-18-rust-1.78.0-nightly.so
needs to be present in $(rustc --print target-libdir)
. It seems Edit: Forgot it is supposed to be in llvm-tools-preview, not in rustc-dev.libLLVM-18-rust-1.78.0-nightly.so
is now entirely missing from rustc-dev.
By the way libLLVM-18-rust-1.78.0-nightly.so
has never been a symlink afaik. Instead a separate copy was shipped in both the rustc and llvm-tools-preview components. The first for rustc itself to link against and the second for user code to link against. Just like we ship librustc_driver.so
in both the rustc and rustc-dev components. Both copies just happened to be identical.
from rust.
I just looked at the latest nightly and libLLVM-18-rust-1.78.0-nightly.so
is present in both $(rustc --print sysroot)/lib
and $(rustc --print target-libdir)
as it should and librustc_driver.so
has the correct DT_NEEDED
filename for it. libLLVM-18-rust-1.78.0-nightly.so
also has libLLVM-18-rust-1.78.0-nightly.so
as DT_SONAME
.
from rust.
I got confused about when #121395 got merged. I though it was already included in the latest nightly. I just looked at the llvm-preview component for that PR and it includes libLLVM.so.18.1-rust-1.78.0-nightly
rather than the libLLVM-18-rust-1.78.0-nightly.so
that is expected. Is it possible to rename it back and patch up the DT_SONAME
? There is no way to link against libLLVM.so.18.1-rust-1.78.0-nightly
with all linkers without a symlink, while for libLLVM-18-rust-1.78.0-nightly.so
that is not a problem.
from rust.
Why can't rustc_driver include -lLLVM-18.1-rust-1.78.0-nightly
?
from rust.
Because the linker will prefix the name with lib
and postfix it with .so
, so -lLLVM-18.1-rust-1.78.0-nightly
would cause the linker to look for libLLVM-18.1-rust-1.78.0-nightly.so
rather than libLLVM.so.18.1-rust-1.78.0-nightly
. (notice the different location of .so
) The only portable way to link against libfoo.so.x.y.z
is to create a symlink from libfoo.so
to libfoo.so.x.y.z
, set the SONAME
to libfoo.x.y.z
and then tell the linker to link against libfoo.so
. But on Windows we can't use symlinks. While on linux -l:/path/to/libfoo.x.y.z
is possible, this is not the case on most other targets.
from rust.
This is because llvm-config returns something like -lLLVM-18 and the symlink is needed to resolve that to -l:libLLVM.so.18.1
If llvm-config inherently relies on a symlink, how do they support Windows?
Windows doesn't have a concept of sonames, so I don't think a symlink would be used there. I don't know what the dylib is called there. We use static linking there anyway. So I think we wouldn't actually ship a symlink on anything but Linux.
More problematic is the other part of this comment:
rust/src/bootstrap/src/utils/tarball.rs
Lines 346 to 347 in 1090205
If rustup-toolchain-install-master can't deal with symlinks, that's also a problem for Linux.
from rust.
So I think we wouldn't actually ship a symlink on anything but Linux.
Any platform except Windows you mean?
from rust.
klint CI also starts failing with this exact same issue.
from rust.
So to summarize, I see a few options here:
- Undo the library name change by carrying a patch in our LLVM fork. Implies carrying a long-term patch to LLVM, though a fairly small one.
- Undo the library name change after the fact. As @bjorn3 pointed out, this isn't just a matter of moving the files, we also have to fix up the DT_SONAME, which doesn't appear straightforward. Would require adding a dependency on patchelf?
- Ship the symlink in rustup components like rustc-dev, on Linux only. May require updates to tooling like rustup-toolchain-install-master.
- Change the way rustc links against upstream dylib dependencies, by resolving library names. Requires replicating linker search path logic and may break other other things. Not sure if this is possible.
- Same as 4, but only do this in rust_llvms build.rs. Again, not sure about the consequences, but at least this is more limited.
I think 3 is probably ideal long term, but depends on just how broken symlink support is right now. Is this a matter of "we'll copy the file instead of symlinking" or "rustup-toolchain-install-master is going to abort"? Maybe @Mark-Simulacrum knows.
The easiest fix would be 1.
from rust.
There's another fix, which is to ship a libLLVM-18-rust-1.78.0-nightly.so
with content
INPUT(libLLVM.so.18.1-rust-1.78.0-nightly)
from rust.
Oooh, that's a great idea!
from rust.
I got klint CI working with the fix: Rust-for-Linux/klint@277cb26
from rust.
Option 6, convince LLVM that this way of naming libraries is a bad idea and causes problems?
I can't tell to what extent this is caused by Rust idiosyncrasies vs LLVM doing something strange.
There's another fix, which is to ship a libLLVM-18-rust-1.78.0-nightly.so with content
Wait, a .so file can be just a text file...?!?
from rust.
Option 6, convince LLVM that this way of naming libraries is a bad idea and causes problems?
I can't tell to what extent this is caused by Rust idiosyncrasies vs LLVM doing something strange.
Basically, LLVM used to do it's own thing here, and now follows the standard convention for shared object naming on Linux. This turns out to be quite inconvenient for us, but I don't think it would be reasonable to ask LLVM to undo the change. (Though, imho, this change really should not have been done in an rc3 release...)
There's another fix, which is to ship a libLLVM-18-rust-1.78.0-nightly.so with content
Wait, a .so file can be just a text file...?!?
It can be a linker script. Somewhat unusual, but things like libc.so or libcxx.so are often linker scripts.
from rust.
Well, if that linker script solution works that would be great. :)
from rust.
This is now beginning to block Miri development. If someone reading along knows enough about linkers to approve #121967 that would be great. :)
from rust.
Related Issues (20)
- ICE: `node HirId cannot be placed in TypeckResults with hir_owner DefId` HOT 2
- Tracking Issue for __rust_no_alloc_shim_is_unstable HOT 2
- Crater runs for 1.78 HOT 21
- HashSet<K, V> has confusing errors thanks to lax BuildHasher bounds HOT 1
- link to private item of proc-macro crate leads to compiler panic
- rustc interrupted by SIGSEGV HOT 2
- Closure captures not indicated in borrow check error
- x.py: Test `cargo miri test` for doctests HOT 12
- GNU Hurd compilation failure: expected `SystemTime`, found `timespec` HOT 10
- x.py should check host's version of libstdc++ before defaulting to download-ci-llvm HOT 1
- ICE: CFI: `typeid_for_instance: couldn't get fn_abi of instance Layout(ReferencesError(ErrorGuaranteed(())))` HOT 2
- Using derive(PartialEq) on an enum with a variant that accepts(Box<dyn SomeTrait>) causes cryptic build error. HOT 1
- static_mut_refs: Need edition migration guide documentation HOT 3
- static_mut_refs: Should the lint cover hidden references? HOT 10
- static_mut_refs: Automatic edition migration HOT 6
- CFI: segfault with debuginfo=2 since llvm18 HOT 3
- Postfix match: Unexpected `unused_parens` warning when LHS contained parenthesis
- field "never read" warning for field that is used by derive(Debug) HOT 7
- Recursive async functions don't internally implement auto traits
- using mold results into panic HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rust.