cchardet build error (M1) about dotrun HOT 14 CLOSED

We might encounter this issue many times on aarch64, because dependencies can require different system packages to be built.

@steverydz had a similar issue on snapcraft where we had to roll back a dependency version as well: canonical/snapcraft.io#3886.

from dotrun.

Yeah let's reopen this until we are able to support cchardet in dotrun on M1

from dotrun.

Is there any update on this? We're still being blocked by this issue on maas.io.

from dotrun.

BTW, as a workaround - manually removing cchardet==2.1.7 from requirements.txt allows to complete the build successfully.

from dotrun.

cchardet either needs to publish a binary wheel for aarch64 to support the M1 (see PyYoshi/cChardet#72), or the build environment needs to install python-dev so it can be compiled (TIL dotrun sets that up)

from dotrun.

We might encounter this issue many times on aarch64, because dependencies can require different system packages to be built.

@steverydz had a similar issue on snapcraft where we had to roll back a dependency version as well: canonical-web-and-design/snapcraft.io#3886.

That should be wholly unnecessary - see the fix we did for maas.io in canonical/maas.io#707

from dotrun.

@sparkiegeek I think you are considering my previous comment to be about this specific issue, I am saying that dotrun might be missing system libraries on that architecture in order to build python wheels for libraries that don't provide it.

It's great that you were able to fix this issue by making it optional on aarch64 I think we should do that on snapcraft.io. but that is not fixing the fact that dotrun is not able to work with that library in this architecture. I think you PR fixed the issue on MAAS repo canonical/maas.io#698 but not this one.

from dotrun.

@sparkiegeek I think you are considering my previous comment to be about this specific issue, I am saying that dotrun might be missing system libraries on that architecture in order to build python wheels for libraries that don't provide it.

Sorry the "wholly unnecessary" was in response to reverting a change on snapcraft.io because it fails to run on an M1 (which the website never will in production). I think it exposes a dangerous precedent to favour dev environments over production. Anyway, hopefully other sites can use the same trick for Python packages which are optional for speedups (like cchardet)

It's great that you were able to fix this issue by making it optional on aarch64 I think we should do that on snapcraft.io. but that is not fixing the fact that dotrun is not able to work with that library in this architecture. I think you PR fixed the issue on MAAS repo canonical-web-and-design/maas.io#698 but not this one.

Correct.

dotrun should probably just install python-dev in the environment which it installs Python packages in?

from dotrun.

It does

https://github.com/canonical-web-and-design/dotrun/blob/main/snap/snapcraft.yaml#L50

I suspect (from 0 investigation) the problem is that python-dev isn't able to compile the relevant binary on that specific architecture so it just skips it. Or something.

from dotrun.

It does

https://github.com/canonical-web-and-design/dotrun/blob/main/snap/snapcraft.yaml#L50

I suspect (from 0 investigation) the problem is that python-dev isn't able to compile the relevant binary on that specific architecture so it just skips it. Or something.

Ah, good point.

I think the problem lies here:

aarch64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Isrc/ext/uchardet/src -I/home/parallels/parallels-code/maas.io/.venv/include -I/usr/include/python3.8 -c src/cchardet/_cchardet.cpp -o build/temp.linux-aarch64-3.8/src/cchardet/_cchardet.o

Notice the -I/usr/include/python3.8 which needs to be rewritten for the snap to be $SNAP/usr/include/python3.8

from dotrun.

Thanks Adam, that's a helpful lead. I imagine someone on the right hardware could test this with something like snap run --shell dotrun and then trying pip3 install cchardet and also maybe running that above command manually, and trying to replacing with $SNAP as you suggest. If it is that, I wonder if setting PYTHONEXECUTABLE or something might help.

But it does seem odd that this problem would only happen in aarch64. I'm sure plenty of modules that need python-dev to build have worked perfectly fine on other architectures. Doesn't seem like that would be true if it was just a path problem.

from dotrun.

But it does seem odd that this problem would only happen in aarch64. I'm sure plenty of modules that need python-dev to build have worked perfectly fine on other architectures. Doesn't seem like that would be true if it was just a path problem.

A lot of packages are Python only, a lot of the ones that aren't build and publish binary wheels, you're just left with a few that fall through those and need to compile things at install time.

I would argue the real issue is the lack of binary wheels from cChardet (AKA PyYoshi/cChardet#72 )

Doubt PYTHONEXECUTABLE will help. See https://docs.python.org/3/install/index.html#alternate-installation-unix-the-prefix-scheme for possible investigation

from dotrun.

A lot of packages are Python only, a lot of the ones that aren't build and publish binary wheels, you're just left with a few that fall through those and need to compile things at install time.

Yeah but I'm like 90% sure we've got dependencies that need compiling that have worked fine in dotrun. But it's easy for me to check. I'll do that when I next get back to a terminal.

from dotrun.

This shouldn't be an issue with dotrun v2

from dotrun.