Add an option for static executables about dwarfs HOT 24 CLOSED

I've tried to use the flags you've indicated, however unfortunately OpenSUSE fails to provide a static variant of Boost. (I.e. they are missing the .a variants, and at least for Boost there isn't a -devel-static package.)

Thus it would be lovely if besides the source code bundle you could also publish static linked variants of the three important tools mkfs, fsck and the FUSE daemon.

from dwarfs.

Thanks for the effort, it's getting closer, however I'm getting the following error:

./dwarfs-0.2.3-Linux/bin/mkdwarfs: /lib64/libpthread.so.0: version `GLIBC_2.30' not found (required by ./dwarfs-0.2.3-Linux/bin/mkdwarfs)
./dwarfs-0.2.3-Linux/bin/mkdwarfs: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by ./dwarfs-0.2.3-Linux/bin/mkdwarfs)
./dwarfs-0.2.3-Linux/bin/mkdwarfs: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by ./dwarfs-0.2.3-Linux/bin/mkdwarfs)
./dwarfs-0.2.3-Linux/bin/mkdwarfs: /lib64/libc.so.6: version `GLIBC_2.27' not found (required by ./dwarfs-0.2.3-Linux/bin/mkdwarfs)
        linux-vdso.so.1 (0x00007ffec89cb000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f113b4b0000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f113b2ac000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f113af74000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f113abb9000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f113b6cf000)

Fortunately I can explain what happens here, because while I was playing with patchelf I managed to solve this issue, although it's not straight forward.

The glibc version you are using is 2.30, and thus some its functions that were introduced only in 2.30 are being used, the same for the other error lines. On the Linux I've tested (OpenSUSE Leap 15.2, latest "stable" let's say), there is only glibc 2.26, thus starting the tool won't work.

There are two potential solutions:

• (preferably) find a way to tell gcc / clang to use older "interface", say according to pkgs.org, 2.12 (which covers CentOS 6) or 2.17 (which covers CentOS 7); (although I have no idea how;)
• alternatively include in the tar.gz bundle also the /lib64 files as identified by ldd, including /lib64/ld-linux-*; (then by using patchelf with --set-rpath and --set-interpreter one can "patch" the executables to use the proper libraries;)

However I think that even with the "hack" described above the deployment is much easier than hunting all the Boost dependencies. :)

I'll experiment more with this subject and let you know.

Ciprian.

from dwarfs.

Indeed it fails due to the mismatch of ld-linux-*.so.

Additionally you need to do the following:

• use readlink -e /lib64/ld-linux-x86-64.so.2 to find it's real name; (on my system it is /lib64/ld-2.32.so;)
• copy your ld-linux-x86-64.so.2 on the target machine under the name /usr/local/lib64/ld-2.32.so (or anything else);
• use patchelf --set-interpreter /usr/local/lib64/ld-2.32.so .../mkdwarfs;

Then try to execute it. For me it worked without an issue.

from dwarfs.

In case of RPATH, the ${ORIGIN} represents the folder where the executable was found.

So for example if one deploys mkdwarfs under /whatever/bin/mkdwarfs, then by using ${ORIGIN}/../share/lib would first search for libraries under /whatever/share/lib. Without the ${ORIGIN} I think it would search the libraries relative to the current folder (i.e. cwd)?

Basically using RPATH and ${ORIGIN} is a nice way to create relocatable executables that depend on .so libraries not present on the target system. (They work as long as one keeps together the binaries and the libraries in the same "relative" relation.)

Regarding the --set-interpreter: on your Ubuntu and Gentoo they seem to work because perhaps you already have installed a "new enough" glibc that is compliant with your build machine.

Thus I had to use it to force the interpreter to the one you've provided in your lib folder.

Now regarding this "binary distribution" I'm suggesting approaching things like this:

• don't touch patchelf --set-interpreter by default, in the hopes that glibc is new enough on the target machine;
• however do provide in the lib folder of your binary distribution, the ld-*.so that you've used on your build machine, and instruct users to patchelf --set-interpreter /final-location/lib/ld-... in case it is needed.

from dwarfs.

FYI, I can build fully static binaries now. I'm just not sure if I can do it with CMake...

$ ldd dwarfs
    not a dynamic executable
$ ./dwarfs
00:27:54.176384 dwarfs (0.2.3)
dwarfs (c) Marcus Holland-Moritz
[...]

from dwarfs.

If you get a chance, could you give the binaries in dwarfs-0.2.4-Linux.tar.gz a quick try?

They're fully static now, and appear to work fine on an old Ubuntu 16 VM. I've also included both FUSEv2 (dwarfs2) and FUSEv3 (dwarfs) versions of the FUSE driver, as old distributions (like Ubuntu 16) don't ship FUSEv3 yet.

from dwarfs.

I've just tried the static binaries (actually only the mkdwarfs for the moment) and they seem to work.

(I'm resuming my CDNJS experiment. Hopefully it will work this time.) :)

from dwarfs.

Just wanted to comment on my previous advice:

DO NOT PUT ANY .so files under /usr/local/lib64, /usr/lib64 or /lib64, or anywhere else listed under /etc/ld.so.conf!

Else you'll break your system, because the next time the /etc/ld.so.cache is rebuilt it will pick up the new libraries and it will break each and every dynamic tool on your system. If this happens, reboot with a live-cd and just remove the /etc/ld.so.cache file.

The correct solution would be to shove them somewhere where the default system won't search for them, for example under /opt/local/dwarfs/lib64.

Just wanted to mention there are no other risks attached by using my proposed solution, except if you put them in a place that are picked by the system. One can safely put the executables in the PATH without issues.

from dwarfs.

If you do any further experiments, could you please play with the v0.3.0 release candidate?

I'm just using v0.3.0 to compress a large personal email archive (~3 M emails, ~55 GiB, most emails are duplicated and only differ in one or two headers, thus ~1.5 actual emails).

from dwarfs.

This is certainly possible and in fact

cmake .. -DCMAKE_EXE_LINKER_FLAGS="-static -Wl,-allow-multiple-definition" -DCMAKE_FIND_LIBRARY_SUFFIXES=".a" -DBoost_USE_MULTITHREADED=OFF -DBoost_USE_STATIC_LIBS=ON

will get you somewhere close at least. This works for me until cmake gets to linking the final binaries. For some reason, the options above fail to find some static boost and non-boost libraries that are definitely present alongside the .sos. Anyway, if you

make VERBOSE=1

and manually "fix" the linker command, it seems to work fine:

$ readelf -d mkdwarfs 

There is no dynamic section in this file.

If I figure out a way to make CMake do the right thing here, I'll document this.

from dwarfs.

• Static builds are something you sometimes want when embedding Gentoo. So it might be a good to have BUILD_STATIC Cmake flag

from dwarfs.

I've managed to port my "shared" executables to another system by using patchelf, changing RPATH and the interpreter.

(Initially I've written quite a long post about how this should be done, but my Chrome crashed, and I've lost all the composed text... If someone is interested let me know and I'll document everything once more.)

from dwarfs.

Ok, after fiddling with this a bit more, I can get down to:

mhx@ubuntu20:~/dwarfs/build-static$ ldd mkdwarfs 
    linux-vdso.so.1 (0x00007ffc5c718000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fb1d08ad000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb1d08a7000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fb1d0758000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb1d0566000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fb1d08dc000)
mhx@ubuntu20:~/dwarfs/build-static$ ldd dwarfs
    linux-vdso.so.1 (0x00007ffdec8bf000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f803f4f5000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f803f4ef000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f803f3a0000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f803f1ae000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f803f524000)

Trying to remove even more gets me into really strange linker error and segfault territory.

Please feel free to check out this binary package (which really isn't 0.2.3, more like 0.3.0-pre): dwarfs-0.2.3-Linux.tar.gz

from dwarfs.

Ah, too bad. The quick test I did was to build the stuff on Ubuntu and then try it out on my Gentoo box, which worked without any issues.

from dwarfs.

rpath is a nice idea, but in practice it's a bit disappointing...

mhx@ubuntu16:~/dwarfs-0.2.3-Linux/bin$ ldd mkdwarfs 
    linux-vdso.so.1 =>  (0x00007ffdeaf7f000)
    libpthread.so.0 => ../share/dwarfs/lib/libpthread.so.0 (0x00007f719bd0a000)
    libdl.so.2 => ../share/dwarfs/lib/libdl.so.2 (0x00007f719bd04000)
    libm.so.6 => ../share/dwarfs/lib/libm.so.6 (0x00007f719bbb4000)
    libc.so.6 => ../share/dwarfs/lib/libc.so.6 (0x00007f719b917000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f719bb09000)
mhx@ubuntu16:~/dwarfs-0.2.3-Linux/bin$ ./mkdwarfs 
Segmentation fault (core dumped)

And on Gentoo:

mhx@mate ~/tmp/dwarfs-0.2.3-Linux/bin $ ldd mkdwarfs 
    linux-vdso.so.1 (0x00007ffc5b92a000)
    libpthread.so.0 => ../share/dwarfs/lib/libpthread.so.0 (0x00007f6bb9b4c000)
    libdl.so.2 => ../share/dwarfs/lib/libdl.so.2 (0x00007f6bb9b46000)
    libm.so.6 => ../share/dwarfs/lib/libm.so.6 (0x00007f6bb99f7000)
    libc.so.6 => ../share/dwarfs/lib/libc.so.6 (0x00007f6bb9805000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f6bb9b71000)
mhx@mate ~/tmp/dwarfs-0.2.3-Linux/bin $ ./mkdwarfs 
Inconsistency detected by ld.so: dl-call-libc-early-init.c: 37: _dl_call_libc_early_init: Assertion `sym != NULL' failed!

from dwarfs.

UPDATE before following this procedure read my last comment: #4 (comment)!

OK, so bellow I paste the full snippets that I've used to build and deploy the shared executables, which should work regardless of the existing libraries installed on the remote server (including glibc). I assume the following:

• the build is for x86-64; (else perhaps one needs to change some values;)
• one already has the sources checked out in ./sources;
• I assume that your /lib64/ld-linux-x86-64.so.2 is actually a symlink to lib64/ld-2.32.so; else just change the 2.32 token in the snippets;

Configure and build (without installing). I manually force the -ldl and -lm dependencies else for some reason they are not picked up automatically for mkdwarfs and are loaded as a consequence to the other libraries that depend on them.

mkdir -- ./build

find ./build/ -mindepth 1 -delete

cmake \
        -GNinja \
        -S./sources \
        -B./build \
        -Wno-dev \
        -DCMAKE_BUILD_TYPE=Release \
        -DCMAKE_C_COMPILER=clang \
        -DCMAKE_CXX_COMPILER=clang++ \
        -DCMAKE_EXE_LINKER_FLAGS='-ldl -lm' \
        -DWITH_LUA=OFF \
        -DWITH_TESTS=OFF \
#

cmake --build ./build

Then I prepare my own "independent" package that should be copied under /usr/local.

Prepare the empty folders:

mkdir -- \
        ./package \
        ./package/bin \
        ./package/lib64 \
        ./package/man \
        ./package/man/man1 \
#

find \
        ./package/bin/ \
        ./package/lib64/ \
        ./package/man/man1/ \
        -mindepth 1 \
        -print \
        -delete \
#

Copy the executables:

cp \
        -v \
        -t ./package/bin \
        -- \
        ./build/mkdwarfs \
        ./build/dwarfsck \
        ./build/dwarfs \
#

Identify dynamic libraries, search them locally and copy them to the lib64 folder:

patchelf \
        --print-needed \
        ./package/bin/mkdwarfs \
        ./package/bin/dwarfsck \
        ./package/bin/dwarfs \
| sort -u \
| xargs -r -I {} -- \
    find \
        -L \
        /usr/local/lib64/ \
        /usr/lib64/ \
        /lib64/ \
        -type f \
        -name {} \
        -print \
        -quit \
| sort -u \
| xargs -r -I {} -- \
    cp \
        -v \
        -t ./package/lib64 \
        -- \
        {} \
#

Also copy the ld-linux-*.so loader. I assume its version is 2.32; else change accordingly!

cp \
        -v \
        -T \
        -- \
        /lib64/ld-linux-x86-64.so.2 \
        ./package/lib64/ld.so.2.32 \
#

Set the RPATH on the executables to ../lib64 as relative to the bin folder.

patchelf \
        --set-rpath '${ORIGIN}/../lib64' \
        ./package/bin/mkdwarfs \
        ./package/bin/dwarfsck \
        ./package/bin/dwarfs \
#

Set the interpreter on the executables to our custom ld-linux-*.so. Mind the version number.

patchelf \
        --set-interpreter '/usr/local/lib64/ld.so.2.32' \
        ./package/bin/mkdwarfs \
        ./package/bin/dwarfsck \
        ./package/bin/dwarfs \
#

Copy the man-pages:

find ./build/man \
        -type f \
        -name '*.1' \
        -exec cp -v -t ./package/man/man1 -- {} \; \
#

List the resulting package contents:

find ./package -ls

Deploy on the remote server:

rsync \
        --recursive \
        --checksum \
        --itemize-changes \
        --perms --owner --group \
        --chmod a=rX \
        --chown root:root \
        -- \
        ./package/ \
        root@server:/usr/local/ \
#

from dwarfs.

Okay, I've got it all working in the CMakeLists.txt now. Building is likely only going to work on my Ubuntu VM, though, but that should be good enough to ship a binary package. The attached package works fine both on my Gentoo box and on an old Ubuntu 16.04 VM that I just set up.

Here you go: dwarfs-0.2.3-Linux.tar.gz

from dwarfs.

There is a minor issue with the RPATH in the executables:

• it must be ${ORIGIN}/../share/dwarfs/lib (that is via patchelf --set-rpath '${ORIGIN}/../share/dwarfs/lib' ./bin/mkdwarfs);
• currently it is only ../share/dwarfs/lib which doesn't work;

(The ${ORIGIN} is verbatim. This is the way it should actually appear when printed with patchelf --print-rpath. It should be quoted or escaped so it's not touched by cmake, make or the shell.)

After patching the above, and using patchelf --set-interpreter to hard-code the ld-2.31.so that you've provided, it works on OpenSUSE Leap 15.2.

from dwarfs.

Why is the ${ORIGIN} needed? For some reason it's not needed on my machines? Also, the interpreter should already be set correctly in the binaries?

from dwarfs.

D'oh. Silly me. This makes perfect sense. The reason it worked for me is because I never changed from the binaries' directory.

from dwarfs.

Thanks for the thorough explanation!

from dwarfs.

Yeah, I reckoned there was potential for bad stuff happening which is why I put things under ${prefix}/share/dwarfs in my previous attempts to be as far away from anything else as possible. Anyway, given that fully static binaries seem to work quite well now I don't think I'll go further with the shared libs stuff.

from dwarfs.

Closed by accident...

If you do any further experiments, could you please play with the v0.3.0 release candidate?

from dwarfs.

Static executables are available for the 0.3.0 release.

from dwarfs.