lld doesn't work on NixOS about rust HOT 18 OPEN

Summarizing the discussion from this zulip thread. It's unfortunate but there's nothing we can really do on our side: NixOS' packaging needs to be updated for lld/rust-lld to be executed and to produce working binaries.

This issue is technically a duplicate of open issues in https://github.com/NixOS/nixpkgs/ but we won't close it yet for visibility, so that people can see this summary if they encounter the issue themselves.

@DianQK mentions a possible temporary workaround by patching rustc's lld wrapper: patchelf --set-interpreter $(patchelf --print-interpreter $(which patchelf)) ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/gcc-ld/ld.lld . But our understanding is that while that would fix running rust-lld, it may sometimes still not be enough: lld itself may not always produce working binaries, as mentioned by Nemo157.

Disabling rust-lld with -Zlinker-features=-lld via RUSTFLAGS or .cargo/config.toml, will fall back to the previous default linker. This works and will unblock people temporarily until the packaging issue is fixed downstream.

from rust.

is there a way to override globally and tell cargo which ld.lld to use?

You can tell rustc to not pass the path to the self-contained linker with RUSTFLAGS='-Clink-self-contained=-linker -Zunstable-options', that leaves the rest of the new defaults active so that it will still pass -fuse-ld=lld and expect it to be on the PATH. Or you can add it explicitly to the paths given to gcc: RUSTFLAGS="-Clink-arg=-B$(nix-build '<nixpkgs>' --no-out-link -A pkgs.llvmPackages.bintools)/bin"; I expected this might be enough on its own, but because of the order of the flags passed to gcc the self-contained linker is still given priority.

There's two useful debugging flags when testing this: first enable logging the linker command RUSTC_LOG=rustc_codegen_ssa::back::link=info, second ask the linker to log its version RUSTFLAGS='-Clink-arg=-Wl,-v' which also causes collect2 to log the full path to which ld.lld it has chosen (needs the first enabled to actually see the output).

from rust.

The symlink may be broken the next you time do a rustup update, but .cargo/config.toml profiles should help.

from rust.

related: (same error)
#71515

from rust.

cc @lqd (at least, I know they were driving the "use lld by default" effort a la rust-lang/compiler-team#510 )

presumably this was injected by #124129

from rust.

I don't know NixOS, so I'm not going to be of much help. I believe it has issues with lld in general, for example NixOS/nixpkgs#24744 that could be related, or this could be a duplicate of that issue. Changes such as NixOS/nixpkgs#24744 (comment) could be needed to make both lld and rust-lld work there.

cc may not be able to find lld because the toolchain is incomplete (the "No such file or directory" part of the error) or because nix's configuration prevents it (the execvp part of the error, I guess).

To see if we can rule out the first case, I'd like to know the contents of the directories we tell cc to look for the lld wrappers calling rust-lld:

• /home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin: it should at least contain the rust-lld binary, and the following gcc-ld subfolder.
• and /home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/gcc-ld: it should contain the 3/4 target-specific lld executables.

If the contents of the directories match what we expect, I would guess the issue is more NixOS/nixpkgs#24744.

from rust.

• /home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin: it should at least contain the rust-lld binary, and the following gcc-ld subfolder.

It does:

ls /home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin

gcc-ld  rust-lld

• and /home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/gcc-ld: it should contain the 3/4 target-specific lld executables.

Don't know if that requirement is satisfied:

ls /home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/gcc-ld

ld64.lld  ld.lld  lld-link  wasm-ld

If this issue is nix-specific we should probably make it extra clear what the work-around is until the issue is resolved.

I added this to my ~/.cargo/config.toml:

[target.x86_64-unknown-linux-gnu]
rustflags = ["-Zlinker-features=-lld"] # fix nightly bug

from rust.

The expected executables are there, the toolchain looks complete AFAICT. Again I don't know NixOs, but there are wasm and aarch64 targets that already use rust-lld by default, so I would expect these to fail there, regardless of #124129.

Similarly, we now just do what users did to change the linker, with regular link args like -Clink-arg=-fuse-ld=lld, as well as adding a -B path like /home/notyou/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/gcc-ld. One can emulate what rustc does, and that needs to work, does it?

The way to revert to the default linker like you did is indeed explicitly mentioned in the blog post announcing the change.

It seems less like a nightly bug to me than a known nix-specific issue.

from rust.

How are you installing the nightly? It erroring out at that stage seems like it's not "fixing" the ld.lld binary's link paths. Using oxalica/rust-overlay it manages to correctly execute ld.lld, but then if the binary it builds uses any native dependencies it fails because the linker isn't "fixing" the binaries it builds (which is NixOS/nixpkgs#24744 related, oxalica/rust-overlay#168).

from rust.

If it's via nixpkgs-rustup, I assume this patch needs updating as it doesn't match these new file paths: https://github.com/NixOS/nixpkgs/blob/4cc02342ff90b8484f11e68bdac985837bb956f9/pkgs/development/tools/rust/rustup/0001-dynamically-patchelf-binaries.patch (but that will just get it to the same state as oxalica/rust-overlay, the self-contained lld will execute but not fully work correctly).

from rust.

Nixpkgs issue: NixOS/nixpkgs#312661

from rust.

How are you installing the nightly?

Using rustup.

from rust.

@lqd , is there a way to override globally and tell cargo which ld.lld to use?

After installing ld.lld with method proposed in NixOS/nixpkgs#24744 (comment),
cargo continues to use ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/gcc-ld/ld.lld .

Creating a symlink helps:

rm ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/gcc-ld/ld.lld
ln -s $(which ld.lld) ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/gcc-ld/ld.lld	

from rust.

also readelf -p .comment, e.g.

$ readelf -p .comment target/debug/helloworld

String dump of section '.comment':
  [     0]  Linker: LLD 18.1.6
  [    13]  rustc version 1.80.0-nightly (84b40fc90 2024-05-27)
  [    48]  GCC: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0

and to add a bit to Nemo's explanation, nightly has two things enabled on x64 linux:

• -Zlinker-features=+lld, this instructs rustc to try to use lld in known contexts: when using a c/c++ compiler as the linker driver, we add -fuse-ld=lld. Disabling that deactivates using lld altogether of course.
• -Clink-self-contained=+linker, this instructs rustc to pass the path to "our rust-lld in the sysroot". These are -B paths to target-specific copies of lld-wrapper binaries that call rust-lld (let's gloss over flavors and arguments and more importantly, why). Deactivating only that, can be seen as a fallback to system lld.

from rust.

Using oxalica/rust-overlay it manages to correctly execute ld.lld, but then if the binary it builds uses any native dependencies it fails because the linker isn't "fixing" the binaries it builds (which is NixOS/nixpkgs#24744 related, oxalica/rust-overlay#168).

As a note, the mentioned rust-overlay issue is fixed. rust-overlay now wraps the shipped ld.lld with ld-wrapper from nixpkgs, and the nightly rustc with default flags (invoking bundled ld.lld) should work out-of-box.

from rust.

@oxalica I have added ld-wrapped.sh to rustup based on your PR: NixOS/nixpkgs#314268. I'm not sure if my patch is reasonable.

from rust.

@oxalica I have added ld-wrapped.sh to rustup based on your PR: NixOS/nixpkgs#314268. I'm not sure if my patch is reasonable.

Indeed it works for me.

from rust.

NixOS/nixpkgs#314268 is now available in nixos-unstable. It will soon be available in nixos-24.05 or release-24.05: https://nixpk.gs/pr-tracker.html?pr=319172.

from rust.