Requires numpy>=1.16.0 about gsd HOT 14 CLOSED

To solve this, I just need to rebuild the PyPI wheels with "${PYBIN}/pip" install numpy=${VERSION} --progress-bar=off. We just need to determine what is the correct old version that will be compatible with python 27, 36, 37, and supported on the majority of user systems.

from gsd.

It does seem that using NumPy>=1.16.0 worked, at least for my package.

from gsd.

It is not my intention to require a specific modern version of numpy. This issue refers to the PyPI gsd package, correct?

This is the code that builds the PyPI binaries.

Does numpy guarantee forward ABI compatibility? If so, what version would you recommend that I build gsd against to support the majority of users?

You should be able to force install gsd from source to ensure ABI compatibility with your current numpy version.

from gsd.

@theavey Have you tried the pip install --no-binary option?

from gsd.

I don't think --no-binary will change anything. The resulting gsd wheel does not link to or embed numpy libraries:

▶ ls -l                                                                                                                                                                                                                                          [14:21:20]
total 896
-rwxr-xr-x 1 joaander glotzer 855099 Mar  5 18:16 fl.cpython-37m-x86_64-linux-gnu.so
-rw-r--r-- 1 joaander glotzer  37986 Mar  5 18:14 hoomd.py
-rw-r--r-- 1 joaander glotzer    645 Mar  5 18:14 __init__.py
-rw-r--r-- 1 joaander glotzer  12545 Mar  5 18:14 pygsd.py
▶ ldd fl.cpython-37m-x86_64-linux-gnu.so                                                                                                                                                                                                         [14:21:27]
        linux-vdso.so.1 (0x00007fff35f4f000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f4f335d4000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f4f33410000)
        /usr/lib64/ld-linux-x86-64.so.2 (0x00007f4f33860000)

The ABI incompatibility comes because gsd is compiled with the installed numpy C header file (see numpy/numpy#5888). Objects in that header file are larger than those in older versions of numpy binaries installed. The question is, which is the appropriate old version of numpy to build against to provide the widest compatibility with the PyPI ecosystem.

conda-forge uses 1.9.3 for builds on unix platforms and 1.11.3 on windows (http://conda-forge.org/docs/maintainer/knowledge_base.html#building-against-numpy). The PyPI manylinux project does not appear to offer similar guidance. @bdice @vyasr @klarh - you are more familiar with PyPI, what would you suggest?

Installing gsd on the target platform with --no-binary will work around this issue. But the built wheels should still be correct.

from gsd.

@csadorf, I have not tried pip install --no-binary. It seems that that would significantly slow down my build times, which wouldn't be terrible, but I'd prefer not to use the (free) Travis-CI CPU time if it's not necessary.

It does work for me to get around this by requiring NumPy 1.16.0 (which is okay for me), but that binary compatibility issue is something you could think about. Some users might not be so savvy, and the error message I got was not particularly helpful.

from gsd.

It looks like 1.9.1 is pretty safe, based on https://libraries.io/pypi/numpy/usage?kind=runtime , at least. I have avoided putting binaries up on pypi thus far just to avoid dealing with these sorts of conflicts, but of course it is convenient to have binary packages!

from gsd.

It looks like 1.9.1 is pretty safe, based on https://libraries.io/pypi/numpy/usage?kind=runtime , at least. I have avoided putting binaries up on pypi thus far just to avoid dealing with these sorts of conflicts, but of course it is convenient to have binary packages!

Could you help me following your reasoning for that? I would not necessarily conclude that 1.9.1 is safe based on the data.

Edit: It's not obvious to me why 1.9.1 is safe based on the data.

from gsd.

Actually upon looking more carefully I see that the "other" column is not where parsing the requirement had failed or something, but actually includes lower versions, so maybe it's less clear.

from gsd.

OK, thanks for looking into that. Conda-forge must have a reason for using 1.9 so I will follow suit. 1.9 is almost 5 years old so that should cover most users. Building 1.9 for python3.7 is not as easy as pip install numpy==1.9.3. I followed the conda recipe to cause the code to be re-cythonized to enable support for python 3.7. You have to delete two files to trigger cython, and get code from the git archive, because the pypi source files don't actually include the source necessary to build numpy with cython. https://github.com/conda-forge/numpy-feedstock/blob/numpy19/recipe/build.sh . One also needs to disable PEP517 processing, as by default the wheel build process creates an isolated virtual environment and installs the latest numpy per pyproject.toml.

I'm taking detailed notes here as @bdice and @vyasr may want to take the same steps with freud. Users of both gsd and freud have reported similar issues.

#24 makes the changes to build with numpy 1.9.3. The reported bug will not be fixed until v1.6.2 is released, and then only for wheel installs of v1.6.2 and later.

from gsd.

Yes, this looks to be the same issue as glotzerlab/freud#243 and the solution I proposed was essentially what you are doing, building the wheels with the oldest possible NumPy version. For freud, the oldest Python version we support is 1.10, which introduced some features to the Python API that we take advantage of (adding the axis argument to certain functions or something similar).

Regarding how to specify the appropriate version for building wheels without impacting the installation from source, one solution I was considering for freud was to modify the requirements.txt file, or perhaps to create a duplicate file such as requirements_whl.txt. The CircleCI build script that is generating wheels reads the requirements file to determine what needs to be installed before installing freud, so that would be how I would ensure that wheels get built with the correct versions without putting the requirements into install_requires. The main concern I had with using install_requires was that I wasn't sure which version would be installed by default if requirements were specified with >= rather than ==; if NumPy is not installed, we need to ensure that it pulls the oldest version rather than the newest version available on PyPI as long as it's newer than 1.9.3.

from gsd.

@vyasr I would be curious to see how you use requirements.txt to help with this. I was unable to get pip to successfully install numpy 1.9.3 for python37. I also found that the pip wheel build process ignored the python environment I built and created its own in /tmp based on the packages listed in pyproject.toml. I needed to specify --no-build-isolation --no-use-pep517 to prevent this and force the wheel build to occur with the environment that I prepared.

My goal with install_requires is to solve the problem on the user end, so that a user with e.g. numpy 1.5 installed can install a gsd wheel pip install gsd and it will upgrade numpy to meet the dependency or otherwise produce an appropriate error. I tested and confirm that pip pulls in the most recent version that matches the version specification:

# /opt/python/cp37-cp37m/bin/pip install gsd -f dist
Looking in links: dist
Collecting gsd
Collecting numpy<2,>=1.9.3 (from gsd)
  Downloading https://files.pythonhosted.org/packages/91/e7/6c780e612d245cca62bc3ba8e263038f7c144a96a54f877f3714a0e8427e/numpy-1.16.2-cp37-cp37m-manylinux1_x86_64.whl (17.3MB)

My solution for the wheel building script is to build and install numpy by hand as I noted above. See #24

from gsd.

gsd 1.6.2 on PyPI fixes this issue:

Testing with numpy 1.14.0 (for example):

 ▶ docker run --rm -it -u 0 quay.io/pypa/manylinux1_x86_64
[root@5b5bfb4665ac /]# /opt/python/cp35-cp35m/bin/pip install numpy==1.14.0 --progress-bar=off
Collecting numpy==1.14.0
  Downloading https://files.pythonhosted.org/packages/55/7f/50d7b4e9f3493779edb3cec0a6ccf68090bf95f0a3b8a093fc0d467cc6d5/numpy-1.14.0-cp35-cp35m-manylinux1_x86_64.whl (17.1MB)

Installing collected packages: numpy
Successfully installed numpy-1.14.0
[root@5b5bfb4665ac /]# /opt/python/cp35-cp35m/bin/pip install gsd --progress-bar=off
Collecting gsd
  Downloading https://files.pythonhosted.org/packages/01/4b/74ffc6e3accc6663bfeb23d60bad5ba25b51705b52e83d17c131c7d96470/gsd-1.6.2-cp35-cp35m-manylinux1_x86_64.whl (263kB)

Requirement already satisfied: numpy<2,>=1.9.3 in /opt/_internal/cpython-3.5.7/lib/python3.5/site-packages (from gsd) (1.14.0)
Installing collected packages: gsd
Successfully installed gsd-1.6.2
[root@5b5bfb4665ac /]# /opt/python/cp35-cp35m/bin/python -c 'from gsd import fl'
[root@5b5bfb4665ac /]#

from gsd.

Hmm if pip wheel ignores the existing environment then the requirements.txt solution probably wouldn't work. The way freud is currently built, we install from requirements.txt and then call pip wheel, and IIRC the pip wheel call was using the installed packages rather than reinstalling new ones, so I assumed that I could simply ensure that the oldest NumPy version we wanted to support was installed before the call to pip wheel.

from gsd.