pyodide-build: unable to locate numpy package while building the pymatgen package about pyodide HOT 13 OPEN

Ah from your message above:

would build wheel with unsupported tag ('cp310', 'cp311', 'emscripten_3_1_45_wasm32')

it sounds like you are trying to build with Python3.10 but you need to use Python 3.11. I think that'll fix your trouble. We should have better error reporting for this...

from pyodide.

I am getting the same error when building moleculekit
If you look at the pyproject.toml it includes numpy in the build dependencies

[build-system]
requires = [
  "setuptools",
  "numpy>=1.18.5",
  "Cython>=0.29.21",
  "toml",
  "versioneer[toml]==0.28",
]
build-backend = "setuptools.build_meta"

but when running pyodide-build it removes it from the packages

Downloading xbuild environment                                                                                                          
Installing xbuild environment                                                                                                           
* Creating virtualenv isolated environment...
* Installing packages in isolated environment... (Cython>=0.29.21, setuptools, toml, versioneer[toml]==0.28)

I played around to see if it was maybe reading these packages from elsewhere and removed versioneer from the list and indeed pyodide-build updated that message, so it's reading them from pyproject.toml but for some reason it ignores numpy when it's in that list.

python -m build works perfectly fine

from pyodide.

I dug into the source code.
In the file pypabuild.py which is the modified pypa/build used by pyodide-build, there is a function remove_avoided_requirements which takes as input get_unisolated_packages() + AVOIDED_REQUIREMENTS.
Looking at build_env.py in the get_unisolated_packages() it reads a file /home/user/moleculekit/.pyodide-xbuildenv/xbuildenv/pyodide-root/unisolated.txt which contains numpy in it. So indeed pyodide-build is removing on purpose numpy from our build requirements. I don't understand why though.

from pyodide.

Removing numpy from the unisolated.txt file continues along with the build but eventually crashes with:

AssertionError: would build wheel with unsupported tag ('cp310', 'cp311', 'emscripten_3_1_45_wasm32')

ERROR Backend subproccess exited when trying to invoke build_wheel

Which according to this stackoverflow question from 2 years ago is a bug in python 3.10 and numpy? https://stackoverflow.com/a/66908804
I'm out of ideas

from pyodide.

Hi, Stefan, thanks for sharing your similar error. Hopefully this is addressed soon. As an update, I did manage to make a little more progress.

Since you were able to determine the error is confined to python3.10 and numpy, I went ahead and installed a local copy of python3.11, and all of the requisite packages, hoping to simply repeat the process with the newer python version.

Unfortunately, pymatgen does not have a /home/user/pymatgen/.pyodide-xbuildenv/ directory, but I was eventually able to get it running on pyodide anyways by hosting a local copy of the .tar.gz file from https://pypi.org/project/pymatgen/#files following this procedure: https://pyodide.org/en/stable/usage/loading-custom-python-code.html#from-python. Note, if you do this, you'll need to extract the [package] directory from the untared [package]-X.X.X directory, then retar that into the .tar.gz you use to upload.

This worked for pymatgen, but then I had to go through and find/install all of it's dependencies manually. This led me to needing to repeat the process for another unsupported package, spglib. This package has a .pyodide-buildenv/ directory though. Following your procedure of removing numpy from the /home/user/package/.pyodide-xbuildenv/xbuildenv/pyodide-root/unisolated.txt file, I was able to call pyodide build in the /home/user/package directory, and build a -none-any.whl file.

Unfortunately, now this wheel file is being installed correctly, but is not being seen by pymatgen. I can call micropip.install("./spglib-2.2.0-py3-none-any.whl");, and importing micropip and calling micropip.list() shows spglib is imported, but I still get ModuleNotFoundError: No module named 'spglib' when one of the pymatgen modules calls import spglib.

Does anyone have any thoughts about this?

from pyodide.

Any update on this issue? It's still around in the latest pyodide-build version 0.25.1

from pyodide.

Hi Stefan!
Sort of. It's been a while, so this may be slightly inaccurate, but I never got pymatgen to build with the necessary numpy dependency. Instead, I noticed that I didn't intend to use the portion of pymatgen that required numpy, so I edited my local copy of pymatgen to remove the parts that required numpy, essentially removing functionality from pymatgen. While this is a terrible practice for a plethora of reasons, it did work...
Best of luck!

from pyodide.

I guess we should look into this @ryanking13...

from pyodide.

Oh, thanks for the ping @hoodmane. I must have missed this issue.

In the file pypabuild.py which is the modified pypa/build used by pyodide-build, there is a function remove_avoided_requirements which takes as input get_unisolated_packages() + AVOIDED_REQUIREMENTS.
Looking at build_env.py in the get_unisolated_packages() it reads a file /home/user/moleculekit/.pyodide-xbuildenv/xbuildenv/pyodide-root/unisolated.txt which contains numpy in it. So indeed pyodide-build is removing on purpose numpy from our build requirements. I don't understand why though.

This is intended. WebAssembly is a 32-bit environment, so if you install numpy from PyPI when you build, some headers will be incorrect because their values are set based on a 64-bit system. So instead of pulling numpy from PyPI, we redirect it to the numpy we built for WebAssembly to handle it, but I think something was going wrong and this wasn't working properly.

from pyodide.

@ryanking13 I see, but the issue is that my package contains some cython modules and that cython code uses numpy types so we need to have numpy installed to be able to compile the cython code. This used to work in the past as I was able to correctly build and use my packages through pyodide including the cython code.

from pyodide.

@stefdoerr Are you building with Python 3.11?

from pyodide.

I just successfully built moleculekit with the following:

git clone https://github.com/emscripten-core/emsdk --depth 1
./emsdk/emsdk install 3.1.46
source ./emsdk/emsdk_enc.sh
python3.11 -m venv .venv
source .venv/bin/activate
pip install pyodide-build
pyodide build

so it is at least possible to build pymatgen. It would help if you could give more precise information about what you did for us to reproduce it.

from pyodide.

Oh well.. that fixed it 😄 Thanks for the help! But yes that error is really not very helpful, I thought it meant that it had trouble with both python 3.10 and 3.11 not that I should switch from 3.10 to 3.11.

from pyodide.