pypi wheel builds? about gemmi HOT 20 CLOSED

It's indeed much easier with cibuildwheel.
I just made a new release and uploaded 31 wheels:
https://pypi.org/project/gemmi/0.4.4/#files
All were built in Github Actions:
https://github.com/project-gemmi/gemmi/runs/1703771115

from gemmi.

I've uploaded Windows wheels for Python 3.6-3.8:
https://pypi.org/project/gemmi/0.4.1/#files

They were built on AppVeyor:
https://ci.appveyor.com/project/wojdyr/gemmi/builds/34355010
I was going to build also wheels for Python2.7, but it didn't work. I suppose I'd need to setup environment for VC++ or use a different worker image. Anyway, it's probably not important due to Python2 EOL.

from gemmi.

Now I added also Linux wheels:
https://pypi.org/project/gemmi/0.4.2/#files
They were built in docker image quay.io/pypa/manylinux2010_x86_64

from gemmi.

Looks great! I'll give it a try soon. Thanks!

from gemmi.

That would be very nice. I could use it, too.

from gemmi.

I suppose it'd be a significant work to set it up. If I'm wrong and it's relatively easy to do then yes, why not.

BTW, we have conda-forge builds maintained by Matt Swain.

from gemmi.

I agree this would be very useful for speeding up CI for my project as well -- building gemmi from source is currently the rate-limiting step. To my knowledge, this is a very straight-forward thing to add. When building your package for release, you can run:

python setup.py sdist bdist_wheel
twine upload dist/*

This should create 3 files in the dist directory: a tar.gz, a .whl, and a .egg, and upload them to your PyPI channel for gemmi. The gemmi PyPI page currently only has a tar.gz, so I think all that is needed it so add the bdist_wheel command to the setup.py call.

from gemmi.

Yes, that should be sufficient unless you need different binary builds for different python versions. (I don't know anything about the gemmi build process)

from gemmi.

@JBGreisman It's not that simple for C++ extensions.
@Anthchirp is correct, number of wheels needed for each supported python version/target OS/architectures. For example look at numpy pypi download page to realize the amount of work.

Note also that default bdist_wheel you get on linux is not manylinux.

from gemmi.

good point -- I had forgotten about supporting multiple Python versions / OS / architectures.

from gemmi.

Looking at files from conda forge, they have 9 packages: python 3.6, 3.7 and 3.8 for Linux, Mac, Windows (all 64-bit).

For Linux, I suppose it needs to be built on the oldest supported Linux system to be portable (more specifically, the versions of glibc and C++ standard library would need to be old).

from gemmi.

I think the situation is worse for pypi than for conda since bare pip lives in uncontrolled evnironment. Conda builds it's own python and other libraries so environments are under control and its number is quite low. I think it's almost impossible to make a pypi wheels for every combination exist. A meaningful compromise would be support of wheels on major systems and fall back to build-from-source on others.

from gemmi.

That is correct, which is what manylinux1, manylinux2010, manylinux2014 aim to address. I don't know what your current build/deploy process is, but the linked project has docker images that can be used to build wheels on travis as well as a link to a demo project.

from gemmi.

Yes, that's the work I'd like to avoid: setting it up separately for each system and maintaining this setup.
I have CI builds on Travis and AppVeyor and I suppose the wheels could be built there, but it'd still be an additional thing to setup and maintain.

I found an interesting project: conda-press - it creates wheels from conda packages. This would be the way to go, but apparently this project was actively developed only for half a year in 2019. From looking at the issues it doesn't support python3.8 packages yet.

from gemmi.

Great, thank you! That is really helpful.

from gemmi.

Using the new wheels reduces our Linux testing time from 5 minutes 55 seconds to 0 minutes 55 seconds. Massive impact. Thank you!

from gemmi.

Linux builds are a big help! Thanks, @wojdyr

from gemmi.

I suppose issue is not considered [completely] solved, so let me share my recent finding.

The cibuildwheel package can build wheels for wide range of platforms / architectures / python versions on many CI platforms. The list of examples contains pybind11-based project.

In fact, https://github.com/pybind/python_example has tested configuration files for Appveyor, Travis and Github-Actions.

from gemmi.

The latest cibuildwheels defaults to manylinux2014 image for Linux builds, previously it was manylinux2010.
If anyone uses wheels on Linux distributions older than RHEL7 please let me know.

from gemmi.

I'm wondering if it's time to stop testing and building wheels for Python 2.7.
If anyone still needs versions older than 3.6 please let me know.

from gemmi.