Hello, I'm asking for arm64 architecture support. Is this possible ?

Sad, the build failed almost at the end <div class="snippet-clip

arm64 arch support,about maresb/docker-build-s3fs

Comments (12)

NoxInmortus commented on June 17, 2024 1

Some hosting solutions offers arm64 architecture, it is generally cheaper (because the hardware is cheaper), and I have several of thoses running at Scaleway, nothing fancy.

Almosted finished to build with docker buildx for arm64 btw

from docker-build-s3fs.

maresb commented on June 17, 2024 1

Ooh, thanks so much for the info! I've had a lot of distractions come up over the past several weekends, but this is still on my mind. I will definitely have a look.

from docker-build-s3fs.

maresb commented on June 17, 2024

Hi there! Glad to see someone else is using this.

Honestly this hadn't crossed my mind. Is this something I'd be able to test with my raspberry pi?

In theory it should be possible:
https://launchpad.net/ubuntu/xenial/arm64/s3fs/1.79+git90-g8f11507-2

I'm not really a developer, so I don't know so much about cross-compiling. It might be as simple as substituting the amd64 .deb for the arm64. If you want to try and figure it out I'd be happy to get a pull request.

I won't have any time to look at this until this weekend at the very earliest.

from docker-build-s3fs.

NoxInmortus commented on June 17, 2024

Hey,

As compiling is always a pain in the a**, i'm always looking for alternative solution, glad to found yours :)

Raspberry pi is arm and not arm64 so i'm afraid you wont be able to do that from scratch.
It does not need crosscompiling, after reading your repo, I think it may works without any changes, just using docker buildx (multiarch) feature.

I started myself with this : https://mirailabs.io/blog/multiarch-docker-with-buildx/

Will try to build your dockerfile myself if I find some free time soon

from docker-build-s3fs.

maresb commented on June 17, 2024

Cool, thanks!!! Do you mind if I ask what your use case is? What sort of hardware do you have which runs arm64?

from docker-build-s3fs.

NoxInmortus commented on June 17, 2024

Sad,

the build failed almost at the end

=> ERROR [15/17] RUN 060-build-package.sh                                                                                                            596.7s 
------
 > [15/17] RUN 060-build-package.sh:
#20 0.257 ++ find . -maxdepth 1 -name 's3fs-fuse-*' -type d
#20 0.325 + PACKAGE_DIR=./s3fs-fuse-1.86+git-v1.86
#20 0.331 + cd ./s3fs-fuse-1.86+git-v1.86
#20 0.334 + sed -i s/libcurl4-gnutls-dev/libcurl4-openssl-dev/g debian/control
#20 0.424 + sed -i s/--with-gnutls/--with-openssl/g debian/rules
#20 0.498 + '[' '2020-02-16 18:00:00' '!=' NONE ']'
#20 0.508 ++ date -u +%s
#20 0.558 ++ date -u '--date=2020-02-16 18:00:00' +%s
#20 0.601 + '[' 1585590159 -lt 1581876000 ']'
#20 0.609 ++ date -u -R '--date=2020-02-16 18:00:00'
#20 0.654 + sed -i '0,/>  / s/>  .*/>  Sun, 16 Feb 2020 18:00:00 +0000/g' debian/changelog
#20 0.726 + find . -exec touch -m -d '2020-02-16 18:00:00' '{}' +
#20 1.115 + debuild -b -uc -us
#20 4.360  dpkg-buildpackage -rfakeroot -us -uc -ui -b
#20 6.670 dpkg-buildpackage: info: source package s3fs-fuse
#20 6.670 dpkg-buildpackage: info: source version 1.86+git-v1.86-2
#20 6.670 dpkg-buildpackage: info: source distribution UNRELEASED
#20 6.670 dpkg-buildpackage: info: source changed by Ben Mares <[email protected]>
#20 7.931  dpkg-source --before-build s3fs-fuse-1.86+git-v1.86
#20 7.931 dpkg-buildpackage: info: host architecture arm64
#20 12.15  fakeroot debian/rules clean
#20 13.67 dh clean
#20 15.52    dh_clean
#20 16.84  debian/rules build
#20 16.92 dh build
#20 18.52    dh_update_autotools_config
#20 19.72    dh_autoreconf
#20 64.36 configure.ac:30: installing './compile'
#20 64.40 configure.ac:26: installing './config.guess'
#20 64.46 configure.ac:26: installing './config.sub'
#20 64.51 configure.ac:27: installing './install-sh'
#20 64.56 configure.ac:27: installing './missing'
#20 65.61 src/Makefile.am: installing './depcomp'
#20 65.86 parallel-tests: installing './test-driver'
#20 67.02    debian/rules override_dh_auto_configure
#20 67.09 make[1]: Entering directory '/home/deb/s3fs-fuse-1.86+git-v1.86'
#20 67.09 dh_auto_configure -- --with-openssl
#20 68.22       ./configure --build=aarch64-linux-gnu --prefix=/usr --includedir=\${prefix}/include --mandir=\${prefix}/share/man --infodir=\${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --disable-silent-rules --libdir=\${prefix}/lib/aarch64-linux-gnu --libexecdir=\${prefix}/lib/aarch64-linux-gnu --disable-maintainer-mode --disable-dependency-tracking --with-openssl
#20 72.24 configure: WARNING: unrecognized options: --disable-maintainer-mode
#20 73.93 checking build system type... aarch64-unknown-linux-gnu
#20 74.49 checking host system type... aarch64-unknown-linux-gnu
#20 74.50 checking target system type... aarch64-unknown-linux-gnu
#20 74.50 checking for a BSD-compatible install... /usr/bin/install -c
#20 74.82 checking whether build environment is sane... yes
#20 75.22 checking for a thread-safe mkdir -p... /bin/mkdir -p
#20 75.28 checking for gawk... no
#20 75.29 checking for mawk... mawk
#20 75.29 checking whether make sets $(MAKE)... yes
#20 75.70 checking whether make supports nested variables... yes
#20 76.05 checking for g++... g++
#20 76.84 checking whether the C++ compiler works... yes
#20 78.47 checking for C++ compiler default output file name... a.out
#20 78.51 checking for suffix of executables... 
#20 80.15 checking whether we are cross compiling... no
#20 82.56 checking for suffix of object files... o
#20 83.69 checking whether we are using the GNU C++ compiler... yes
#20 84.72 checking whether g++ accepts -g... yes
#20 85.67 checking for style of include used by make... GNU
#20 85.85 checking dependency style of g++... none
#20 85.88 checking for gcc... gcc
#20 86.68 checking whether we are using the GNU C compiler... yes
#20 87.68 checking whether gcc accepts -g... yes
#20 88.54 checking for gcc option to accept ISO C89... none needed
#20 90.92 checking whether gcc understands -c and -o together... yes
#20 92.42 checking dependency style of gcc... none
#20 92.43 checking how to run the C preprocessor... gcc -E
#20 94.29 checking for grep that handles long lines and -e... /bin/grep
#20 94.36 checking for egrep... /bin/grep -E
#20 94.43 checking for ANSI C header files... yes
#20 99.35 checking for sys/types.h... yes
#20 100.5 checking for sys/stat.h... yes
#20 101.7 checking for stdlib.h... yes
#20 103.0 checking for string.h... yes
#20 104.3 checking for memory.h... yes
#20 105.6 checking for strings.h... yes
#20 107.1 checking for inttypes.h... yes
#20 108.3 checking for stdint.h... yes
#20 109.5 checking for unistd.h... yes
#20 110.7 checking sys/xattr.h usability... yes
#20 111.7 checking sys/xattr.h presence... yes
#20 112.0 checking for sys/xattr.h... yes
#20 112.1 checking attr/xattr.h usability... no
#20 113.2 checking attr/xattr.h presence... no
#20 113.7 checking for attr/xattr.h... no
#20 113.7 checking sys/extattr.h usability... no
#20 114.8 checking sys/extattr.h presence... no
#20 115.3 checking for sys/extattr.h... no
#20 115.3 checking s3fs build with nettle(GnuTLS)... no
#20 115.3 checking s3fs build with OpenSSL... yes
#20 115.3 checking s3fs build with GnuTLS... no
#20 115.3 checking s3fs build with NSS... no
#20 115.4 checking for pkg-config... /usr/bin/pkg-config
#20 115.4 checking pkg-config is at least version 0.9.0... yes
#20 115.4 checking for common_lib_checking... yes
#20 115.7 checking compile s3fs with... OpenSSL
#20 115.8 checking for DEPS... yes
#20 116.1 checking for malloc_trim... yes
#20 117.7 checking for library containing clock_gettime... none required
#20 119.2 checking for clock_gettime... yes
#20 120.8 checking pthread mutex recursive... PTHREAD_MUTEX_RECURSIVE
#20 121.9 checking checking CURLOPT_TCP_KEEPALIVE... yes
#20 123.3 checking checking CURLOPT_SSL_ENABLE_ALPN... yes
#20 124.6 checking checking CURLOPT_KEEP_SENDING_ON_ERROR... yes
#20 126.0 checking for git... yes
#20 126.0 checking github short commit hash... v1.86
#20 126.6 checking that generated files are newer than configure... done
#20 126.6 configure: creating ./config.status
#20 129.7 config.status: creating Makefile
#20 130.1 config.status: creating src/Makefile
#20 130.7 config.status: creating test/Makefile
#20 131.3 config.status: creating doc/Makefile
#20 131.9 config.status: creating config.h
#20 132.2 config.status: executing depfiles commands
#20 132.2 configure: WARNING: unrecognized options: --disable-maintainer-mode
#20 132.8 make[1]: Leaving directory '/home/deb/s3fs-fuse-1.86+git-v1.86'
#20 133.0    dh_auto_build
#20 133.9       make -j4 -O
#20 134.1 /usr/bin/make  all-recursive
#20 134.5 Making all in src
#20 224.2 make[3]: Entering directory '/home/deb/s3fs-fuse-1.86+git-v1.86/src'
#20 224.2 g++ -DHAVE_CONFIG_H -I. -I..  -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/aarch64-linux-gnu -I/usr/include/libxml2  -Wdate-time -D_FORTIFY_SOURCE=2  -g -O2 -fdebug-prefix-map=/home/deb/s3fs-fuse-1.86+git-v1.86=. -fstack-protector-strong -Wformat -Werror=format-security -Wall -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2 -c -o string_util.o string_util.cpp
#20 224.2 make[3]: Leaving directory '/home/deb/s3fs-fuse-1.86+git-v1.86/src'
#20 262.8 make[3]: Entering directory '/home/deb/s3fs-fuse-1.86+git-v1.86/src'
#20 262.8 g++ -DHAVE_CONFIG_H -I. -I..  -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/aarch64-linux-gnu -I/usr/include/libxml2  -Wdate-time -D_FORTIFY_SOURCE=2  -g -O2 -fdebug-prefix-map=/home/deb/s3fs-fuse-1.86+git-v1.86=. -fstack-protector-strong -Wformat -Werror=format-security -Wall -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2 -c -o cache.o cache.cpp
#20 262.8 make[3]: Leaving directory '/home/deb/s3fs-fuse-1.86+git-v1.86/src'
#20 384.2 make[3]: Entering directory '/home/deb/s3fs-fuse-1.86+git-v1.86/src'
#20 384.2 g++ -DHAVE_CONFIG_H -I. -I..  -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/aarch64-linux-gnu -I/usr/include/libxml2  -Wdate-time -D_FORTIFY_SOURCE=2  -g -O2 -fdebug-prefix-map=/home/deb/s3fs-fuse-1.86+git-v1.86=. -fstack-protector-strong -Wformat -Werror=format-security -Wall -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2 -c -o s3fs_util.o s3fs_util.cpp
#20 384.2 s3fs_util.cpp: In function 'std::__cxx11::string get_realpath(const char*)':
#20 384.2 s3fs_util.cpp:67:1: internal compiler error: Segmentation fault
#20 384.2  }
#20 384.2  ^
#20 384.2 mmap: Permission denied
#20 384.2 Please submit a full bug report,
#20 384.2 with preprocessed source if appropriate.
#20 384.2 See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
#20 384.2 The bug is not reproducible, so it is likely a hardware or OS problem.
#20 384.2 Makefile:619: recipe for target 's3fs_util.o' failed
#20 384.2 make[3]: *** [s3fs_util.o] Error 1
#20 384.2 make[3]: Leaving directory '/home/deb/s3fs-fuse-1.86+git-v1.86/src'
#20 384.2 make[3]: *** Waiting for unfinished jobs....
#20 516.1 make[3]: Entering directory '/home/deb/s3fs-fuse-1.86+git-v1.86/src'
#20 516.1 g++ -DHAVE_CONFIG_H -I. -I..  -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/aarch64-linux-gnu -I/usr/include/libxml2  -Wdate-time -D_FORTIFY_SOURCE=2  -g -O2 -fdebug-prefix-map=/home/deb/s3fs-fuse-1.86+git-v1.86=. -fstack-protector-strong -Wformat -Werror=format-security -Wall -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2 -c -o s3fs.o s3fs.cpp
#20 516.1 make[3]: Leaving directory '/home/deb/s3fs-fuse-1.86+git-v1.86/src'
#20 517.9 make[3]: Entering directory '/home/deb/s3fs-fuse-1.86+git-v1.86/src'
#20 517.9 g++ -DHAVE_CONFIG_H -I. -I..  -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/aarch64-linux-gnu -I/usr/include/libxml2  -Wdate-time -D_FORTIFY_SOURCE=2  -g -O2 -fdebug-prefix-map=/home/deb/s3fs-fuse-1.86+git-v1.86=. -fstack-protector-strong -Wformat -Werror=format-security -Wall -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2 -c -o fdcache.o fdcache.cpp
#20 517.9 make[3]: Leaving directory '/home/deb/s3fs-fuse-1.86+git-v1.86/src'
#20 596.5 make[3]: Entering directory '/home/deb/s3fs-fuse-1.86+git-v1.86/src'
#20 596.5 g++ -DHAVE_CONFIG_H -I. -I..  -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/aarch64-linux-gnu -I/usr/include/libxml2  -Wdate-time -D_FORTIFY_SOURCE=2  -g -O2 -fdebug-prefix-map=/home/deb/s3fs-fuse-1.86+git-v1.86=. -fstack-protector-strong -Wformat -Werror=format-security -Wall -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2 -c -o curl.o curl.cpp
#20 596.5 curl.cpp: In function 'std::__cxx11::string url_to_host(const string&)':
#20 596.5 curl.cpp:120:1: internal compiler error: Segmentation fault
#20 596.5  }
#20 596.5  ^
#20 596.5 mmap: Permission denied
#20 596.5 Please submit a full bug report,
#20 596.5 with preprocessed source if appropriate.
#20 596.5 See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
#20 596.5 The bug is not reproducible, so it is likely a hardware or OS problem.
#20 596.5 Makefile:619: recipe for target 'curl.o' failed
#20 596.5 make[3]: *** [curl.o] Error 1
#20 596.5 make[3]: Leaving directory '/home/deb/s3fs-fuse-1.86+git-v1.86/src'
#20 596.5 Makefile:400: recipe for target 'all-recursive' failed
#20 596.5 make[2]: *** [all-recursive] Error 1
#20 596.5 Makefile:341: recipe for target 'all' failed
#20 596.5 make[1]: *** [all] Error 2
#20 596.6 dh_auto_build: make -j4 -O returned exit code 2
#20 596.6 debian/rules:6: recipe for target 'build' failed
#20 596.6 make: *** [build] Error 25
#20 596.6 dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2
#20 596.6 debuild: fatal error at line 1152:
#20 596.6 dpkg-buildpackage -rfakeroot -us -uc -ui -b failed

from docker-build-s3fs.

maresb commented on June 17, 2024

Haha, segfault in the compiler! I think that's an achievement. :)

A few thoughts:

Have you checked to see if it's pulling the arm64 .deb file?
Will it compile the code from the .deb file with the original source code (without replacing it with the GitLab source code)?

from docker-build-s3fs.

NoxInmortus commented on June 17, 2024

I tried again, no segfault then. Guess that was my computer doing too much in the same time. Will try to get the deb and test it

EDIT: Can not find out it is saying that

docker cp $id:s3fs-debian-package.tar .
Error: No such container:path: 743f79292a0e28a6a6a7d045d3cc9f2abfbd58d4b27d87425eb31e4d3e28cafa:s3fs-debian-package.tar

EDIT2 : Silly me, that's just and example isnt it ?

there it is :

deb@dcdf946cd907:~$ ls
s3fs-dbgsym_1.86+git-v1.86-2_arm64.ddeb  s3fs-fuse_1.82-1.dsc                    s3fs-fuse_1.86+git-v1.86-2_arm64.buildinfo
s3fs-fuse-1.86+git-v1.86                 s3fs-fuse_1.82.orig.tar.gz              s3fs-fuse_1.86+git-v1.86-2_arm64.changes
s3fs-fuse_1.82-1.debian.tar.xz           s3fs-fuse_1.86+git-v1.86-2_arm64.build  s3fs_1.86+git-v1.86-2_arm64.deb

EDIT3 : It works!
Example (from the link I previously added) :

export DOCKER_CLI_EXPERIMENTAL=enabled
docker run --rm --privileged docker/binfmt:66f9012c56a8316f9244ffd7622d7c21c1f6f28d
docker buildx create --use
docker buildx build -t test --platform=linux/aarch64,linux/amd64,linux/arm .

You can also add linux/aarch64 for raspberry pi :)

What about you testing?

from docker-build-s3fs.

maresb commented on June 17, 2024

Nice work!!! I'll try to test this weekend.

Do you think we could get this properly written up and added?

Also, do you know if there's any way around using --privileged? One of my goals is to keep this build process as safe and transparent as possible, and running that privileged command with the long hex code gives me the creeps.

from docker-build-s3fs.

NoxInmortus commented on June 17, 2024

Well, as such there is not much to add, the docker buildx is an alternative to not having to build directly on the arch you want, so a bit of readme eventually but that's all ?

About the docker run --rm --privileged docker/binfmt:66f9012c56a8316f9244ffd7622d7c21c1f6f28d
I have found this official docker image on several forums or good looking tutorials like the one I gave, but nothing official and it bothers me too, but this is unblocked the unavailable architectures on my laptop. But I do not need it when building with gitlab-ci on my virtual machines tho.

I suggest that you could add the tested architectures on your readme, and try docker buildx multiarch feature by yourself.

from docker-build-s3fs.

maresb commented on June 17, 2024

@NoxInmortus thanks for closing this, but I would like to address this issue at some point. I actually made an attempt at cross-compiling about two weeks ago, but I was not successful. (I just don't have enough experience to be able to properly diagnose things when a build fails.)

I also recently came across a multi-arch version of conda, and I wonder if I could adapt their pipeline. They use something called qemu-user-static, but I'd need to look into it further to understand how it works.

from docker-build-s3fs.

NoxInmortus commented on June 17, 2024

Hello @maresb, hope you're doing fine in those troubled times.

I very recently ended a short project to quickly build some debian package for many differents architectures. I've almost never built packages before, but I finally made it. Looking back on this issue I think it might interrest you if you need some references.

Here is the repository : https://github.com/NoxInmortus/docker-build-pqchecker

I'm using gitlab-runner and docker buildx to build the images.

I'm using this ansible role to install gitlab-runner : https://gitlab.com/guardianproject-ops/ansible-gitlab-runner.git
And this set of task to allow gitlab-runner to run correctly docker buildx :

---
- name: gitlab-runner | Mkdir ${HOME}/.docker/cli-plugins
  file:
    state: directory
    path: /home/gitlab-runner/.docker/cli-plugins
    owner: gitlab-runner
    group: gitlab-runner

- name: gitlab-runner | rm ${HOME}/.bash_logout
  file:
    state: absent
    path: /home/gitlab-runner/.bash_logout

- name: gitlab-runner | Get docker-pushrm cli-plugin
  shell: 'wget -q https://github.com/christian-korneck/docker-pushrm/releases/download/v1.5.1/docker-pushrm_linux_amd64 -O /home/gitlab-runner/.docker/cli-plugins/docker-pushrm'
  args:
    creates: /home/gitlab-runner/.docker/cli-plugins/docker-pushrm

- name: gitlab-runner | Get docker-buildx cli-plugin
  shell: 'wget -q https://github.com/docker/buildx/releases/download/v0.4.2/buildx-v0.4.2.linux-amd64 -O /home/gitlab-runner/.docker/cli-plugins/docker-buildx'
  args:
    creates: /home/gitlab-runner/.docker/cli-plugins/docker-buildx

- name: gitlab-runner | Chown/Chmod docker cli-plugins
  file:
    path: '/home/gitlab-runner/.docker/cli-plugins/docker-{{ item }}'
    owner: gitlab-runner
    group: gitlab-runner
    mode: '0755'
  with_items:
    - pushrm
    - buildx

- name: gitlab-runner | Add gitlab-runner to docker group
  user:
    name: gitlab-runner
    groups:
      - docker
  notify: docker_restart

- name: gitlab-runner | Enable experimental docker features
  copy:
    dest: /etc/docker/config.json
    content: |
      {
      "experimental": true
      }
  notify: docker_restart

- name: gitlab-runner | install qemu-user-static binfmt-support
  apt:
    name: '{{ item }}'
  with_items:
    - qemu-user-static
    - binfmt-support

- name: gitlab-runner | docker buildx create --use --name multiarch
  command: docker buildx create --use --name multiarch
  become: yes
  become_user: gitlab-runner
  args:
    creates: /home/gitlab-runner/.docker/buildx/instances/multiarch

Hope it helps.

from docker-build-s3fs.

arm64 arch support about docker-build-s3fs HOT 12 OPEN

Comments (12)

Related Issues (8)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent