Establish package distribution strategy about crosstool-ng-old HOT 15 OPEN

That said for us the most convenient would be "tarball" way of distributing things though I remember that existing SDK installer script runs patchelf to fix-up RPATH in installed binaries ans so I'm wondering if something similar will be needed for that modular "toolchain" and if so how that's going to be fixed?

In terms of the GNU toolchains (binutils-gdb, gcc), I have taken extra care to minimise shared library dependencies by static-linking almost everything, so practically speaking it should be able to run on all modern Linux distros (RPATH-ing won't be necessary either):

$ ldd arc-zephyr-elf-ld
        linux-vdso.so.1 (0x00007ffe7e34f000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fcfc4147000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fcfc3d56000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fcfc468e000)

$ ldd arc-zephyr-elf-gcc
        linux-vdso.so.1 (0x00007ffda0ecb000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f80ceed4000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f80cf2c5000)

$ ldd arc-zephyr-elf-gdb
        linux-vdso.so.1 (0x00007ffe62bcf000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f9db8e68000)
        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f9db8c42000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f9db88a4000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f9db8685000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9db8294000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f9db9bc7000)

In addition, the GNU toolchains have been configured to support relocatable sysroot (i.e. relative path resolution), so using toolchain tarballs unpacked under an arbitrary path in the user machine will not be a problem.

(gdb) show config
This GDB was configured as follows:
   configure --host=x86_64-build_pc-linux-gnu --target=arc-zephyr-elf
             --with-auto-load-dir=$debugdir:$datadir/auto-load
             --with-auto-load-safe-path=$debugdir:$datadir/auto-load
             --with-expat
             --with-gdb-datadir=/home/runner/output/arc-zephyr-elf/share/gdb (relocatable)
             --with-jit-reader-dir=/home/runner/output/arc-zephyr-elf/lib/gdb (relocatable)
             --without-libunwind-ia64
             --with-lzma
             --without-babeltrace
                             --without-intel-pt
             --with-mpfr
             --without-xxhash
             --without-python
             --without-guile
             --enable-source-highlight
             --with-separate-debug-dir=/home/runner/output/arc-zephyr-elf/lib/debug (relocatable)
             --with-sysroot=/home/runner/output/arc-zephyr-elf/arc-zephyr-elf (relocatable)

("Relocatable" means the directory can be moved with the GDB installation
tree, and GDB will still find it.)

from crosstool-ng-old.

RFC: Side-by-side Installation and Bundled Archive Distribution Format

It may be desirable to install multiple versions of the same package side-by-side (referred to as "SxS installation" hereinafter) in order to facilitate development and testing of the legacy Zephyr versions that are not compatible with the latest SDK packages.

While most package management systems listed above support SxS package installation to some degree, switching among the different versions is not trivial and/or practical because users are responsible for keeping track of the required package versions and (un-)aliasing/shimming each package whenever a version switch is required; note that there are over 10 component packages and it is simply not practical for users to manually manage them.

In order to address this problem, it is necessary to introduce an alternate archive-based distribution format that bundles the development tools that are necessary to build a specific version of Zephyr RTOS (referred to as "bundle" hereinafter).

A bundle distribution shall:

• have a version number that matches the compatible Zephyr RTOS version (e.g. for Zephyr RTOS 2.4, the compatible bundles shall be versioned 2.4.x)
• contain the specific version of development tools that are necessary to build the corresponding Zephyr RTOS version
• have the internal directory and file organisation that is resolvable by the build system of the corresponding Zephyr RTOS version
• be selected by setting ZEPHYR_SDK_TYPE to bundle and ZEPHYR_SDK_INSTALL_DIR to the topmost directory of the bundle (see zephyrproject-rtos/zephyr#25475 (comment))
• contain a CMake package (ZephyrSdkBundle) that can aid the Zephyr RTOS build system in automagically finding the correct bundle to use
• be available for Linux, macOS and Windows operating systems

The target audience for each distribution format is as follows:

• Package distribution
  • Beginners who want something that is quick and easy to set up
  • Zephyr developers who usually work on the master branch
• Bundled archive distribution (for SxS installation)
  • Zephyr developers who work on backports
  • End users whose projects are based on an older version of Zephyr

Note that the bundled archive distribution may be installed alongside the package distribution. This allows developers to use the package distribution for daily development on the master branch, while using the bundled archive distribution for backports and specific projects that are based on the legacy Zephyr versions.

from crosstool-ng-old.

cc @nashif @galak @carlescufi

from crosstool-ng-old.

@stephanosio I think this is worthy of a TSC discussion. I would suggest to @MaureenHelm to add this to the agenda of a TSC meeting soon.

from crosstool-ng-old.

@stephanosio I think this is worthy of a TSC discussion. I would suggest to @MaureenHelm to add this to the agenda of a TSC meeting soon.

Added to the opens list in the TSC meeting minutes. @stephanosio I think the TSC meeting time is pretty late for you, but if you want to join let me know if there's a particular upcoming week that works best.
https://github.com/zephyrproject-rtos/zephyr/wiki/Zephyr-Committee-and-Working-Group-Meetings

from crosstool-ng-old.

I do like that proposal and would just like to add my 2 cents.
We still happen to use bare-metal servers [typically with CentOS on top] internally as the main compute infrastructure and so there're some limitations:

1. There's no Snap installed and I'm pretty sure never will be
2. As users we cannot install apps via packages
   • And as I see .rpm's are not even considered to be built (that's OK we don't want to cover everythig)
   • We don't even want to as we'll need to do it on multiple machines every time we want a newer something

That said for us the most convenient would be "tarball" way of distributing things though I remember that existing SDK installer script runs patchelf to fix-up RPATH in installed binaries ans so I'm wondering if something similar will be needed for that modular "toolchain" and if so how that's going to be fixed?

from crosstool-ng-old.

Well that's understood and I think we use pretty much the same in our prebuilt tools.
But Zephyr's host tools seem to have RPATH pointing to OpenEmbedded's sysroot.
Just now bumped into incomplete SDK installation with not-elfpatched dtc and seeing this:

$ .../tools/zephyr-sdk/0.11.3/sysroots/x86_64-pokysdk-linux/usr/bin/dtc --version
-bash: .../tools/zephyr-sdk/0.11.3/sysroots/x86_64-pokysdk-linux/usr/bin/dtc: /opt/zephyr-sdk/2.2/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2: bad ELF interpreter: No such file or directory

$ ldd .../tools/zephyr-sdk/0.11.3/sysroots/x86_64-pokysdk-linux/usr/bin/dtc
        linux-vdso.so.1 =>  (0x00007ffff7ffd000)
        libc.so.6 => /lib64/libc.so.6 (0x00007ffff7bd0000)
        /opt/zephyr-sdk/2.2/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)

from crosstool-ng-old.

Well that's understood and I think we use pretty much the same in our prebuilt tools.
But Zephyr's host tools seem to have RPATH pointing to OpenEmbedded's sysroot.
Just now bumped into incomplete SDK installation with not-elfpatched dtc and seeing this:

The Zephyr SDK (sdk-ng) has various issues including that. That was one of the motivations for this too actually.

from crosstool-ng-old.

Is there a solution for metapkg on brew? this was something that @beriberikix looked into and we didn't seem to have a good solution here.

from crosstool-ng-old.

We need to figure out how to handle CI for all the platforms configurations (since github doesn't cover all of what we need, for example aarch64 for linux, m1/arm64 for macos).

from crosstool-ng-old.

We need to figure out how to handle CI for all the platforms configurations (since github doesn't cover all of what we need, for example aarch64 for linux, m1/arm64 for macos).

Very good point, how do we handle that right now, is it just via manual test? Or we don't support it at all?

from crosstool-ng-old.

We need to figure out how to handle CI for all the platforms configurations (since github doesn't cover all of what we need, for example aarch64 for linux, m1/arm64 for macos).

Very good point, how do we handle that right now, is it just via manual test? Or we don't support it at all?

We build on linux x86 and arm64/aarch64 on every PR today (via buildkite). I just added x86_64 mac os builds (via github) these are only done after we commit due to the length of time this takes.

from crosstool-ng-old.

We build on linux x86 and arm64/aarch64 on every PR today (via buildkite). I just added x86_64 mac os builds (via github) these are only done after we commit due to the length of time this takes.

So for everything outside of mac os/m1/arm I believe we can utilize AWS, but need to understand what the setup in AWS would look like for windows and macos x86_64. We can probably utilize a similar setup (buildkite or otherwise) on MacOS ARM/M1, just need to see what service we'd utilize to do MacOS ARM/M1 builds.

from crosstool-ng-old.

We should look at adding a zephyr-bossac package as well. (I think that would flush out the hosttools).

from crosstool-ng-old.

Is there a solution for metapkg on brew? this was something that @beriberikix looked into and we didn't seem to have a good solution here.

Yes, see https://github.com/stephanosio/homebrew-zephyr/blob/master/Formula/zephyr-crosstool.rb.

Also stephanosio/homebrew-zephyr#5.

We need to figure out how to handle CI for all the platforms configurations (since github doesn't cover all of what we need, for example aarch64 for linux, m1/arm64 for macos).

In general, it would be best to move away from using GitHub Actions. I found that GitHub Actions is way too unstable for this purpose -- you'd get occasional (more like 2/5 times, actually ...) job cancellations pretty much on a random basis and you have to restart the whole workflow again, until none of the jobs fails and that can take lots of retries. Using GitHub Actions for a complex and lengthy workflow is very frustrating to say the least.

So for everything outside of mac os/m1/arm I believe we can utilize AWS, but need to understand what the setup in AWS would look like for windows and macos x86_64. We can probably utilize a similar setup (buildkite or otherwise) on MacOS ARM/M1,

Yes, AWS+Buildkite sounds good.

For Windows host toolchains, we Canadian-cross compile from Linux so no need for Windows VMs -- we can just use the existing Linux VMs. I also tried Canadian-crossing macOS host toolchains from Linux build machine, but that didn't go well unfortunately.

just need to see what service we'd utilize to do MacOS ARM/M1 builds.

There are M1 hosting services out there but most of them seemed to be out of stock right now. That aside, I think we can potentially Canadian-cross Darwin ARM64 host toolchain from Darwin x86-64 build machine. That requires investigation but I would bet that it works as long as we properly provide the host libraries on the build machine.

from crosstool-ng-old.