Giter VIP home page Giter VIP logo

pytables-wheels's Introduction

Building and uploading pytables wheels

We automate wheel building using this custom github repository that builds on the travis-ci OSX machines and the travis-ci Linux machines.

The travis-ci interface for the builds is https://travis-ci.org/MacPython/pytables-wheels

The appveyor interface for the builds is https://ci.appveyor.com/project/matthew-brett/pytables-wheels

The driving github repository is https://github.com/MacPython/pytables-wheels

How it works

The wheel-building repository:

  • does a fresh build of any required C / C++ libraries;
  • builds a pytables wheel, linking against these fresh builds;
  • processes the wheel using delocate (OSX) or auditwheel repair (Manylinux1). delocate and auditwheel copy the required dynamic libraries into the wheel and relinks the extension modules against the copied libraries;
  • uploads the built wheels to http://wheels.scipy.org (a Rackspace container kindly donated by Rackspace to scikit-learn).

The resulting wheels are therefore self-contained and do not need any external dynamic libraries apart from those provided as standard by OSX / Linux as defined by the manylinux1 standard.

The .travis.yml file in this repository has a line containing the API key for the Rackspace container encrypted with an RSA key that is unique to the repository - see http://docs.travis-ci.com/user/encryption-keys. This encrypted key gives the travis build permission to upload to the Rackspace directory pointed to by http://wheels.scipy.org.

Triggering a build

You will likely want to edit the .travis.yml file to specify the BUILD_COMMIT before triggering a build - see below.

You will need write permission to the github repository to trigger new builds on the travis-ci interface. Contact us on the mailing list if you need this.

You can trigger a build by:

  • making a commit to the pytables-wheels repository (e.g. with git commit --allow-empty); or
  • clicking on the circular arrow icon towards the top right of the travis-ci page, to rerun the previous build.

In general, it is better to trigger a build with a commit, because this makes a new set of build products and logs, keeping the old ones for reference. Keeping the old build logs helps us keep track of previous problems and successful builds.

Which pytables commit does the repository build?

The pytables-wheels repository will build the commit specified in the BUILD_COMMIT at the top of the .travis.yml file. This can be any naming of a commit, including branch name, tag name or commit hash.

Uploading the built wheels to pypi

Be careful, http://wheels.scipy.org points to a container on a distributed content delivery network. It can take up to 15 minutes for the new wheel file to get updated into the container at http://wheels.scipy.org.

The same contents appear at https://3f23b170c54c2533c070-1c8a9b3114517dc5fe17b7c3f8c63a43.ssl.cf2.rackcdn.com; you might prefer this address because it is https.

When the wheels are updated, you can download them to your machine manually, and then upload them manually to pypi, or by using twine. You can also use a script for doing this, housed at : https://github.com/MacPython/terryfy/blob/master/wheel-uploader

For the wheel-uploader script, you'll need twine and beautiful soup 4.

You will typically have a directory on your machine where you store wheels, called a wheelhouse. The typical call for wheel-uploader would then be something like:

VERSION=3.3.0
CDN_URL=https://3f23b170c54c2533c070-1c8a9b3114517dc5fe17b7c3f8c63a43.ssl.cf2.rackcdn.com
wheel-uploader -r warehouse -u $CDN_URL -s -v -w ~/wheelhouse -t macosx tables $VERSION
wheel-uploader -r warehouse -u $CDN_URL -s -v -w ~/wheelhouse -t manylinux1 tables $VERSION

where:

  • -r warehouse uses the upcoming Warehouse PyPI server (it is more reliable than the current PyPI service for uploads);
  • -u gives the URL from which to fetch the wheels, here the https address, for some extra security;
  • -s causes twine to sign the wheels with your GPG key;
  • -v means give verbose messages;
  • -w ~/wheelhouse means download the wheels from to the local directory ~/wheelhouse.

tables is the root name of the wheel(s) to download / upload, and 3.3.0 is the version to download / upload.

In order to use the Warehouse PyPI server, you will need something like this in your ~/.pypirc file:

[distutils]
index-servers =
    pypi
    warehouse

[pypi]
username:your_user_name
password:your_password

[warehouse]
repository: https://upload.pypi.io/legacy/
username: your_user_name
password: your_password

So, in this case, wheel-uploader will download all wheels starting with tables-3.3.0- from http://wheels.scipy.org to ~/wheelhouse, then upload them to PyPI.

Of course, you will need permissions to upload to PyPI, for this to work.

pytables-wheels's People

Contributors

andreabedini avatar francescalted avatar jsancho-gpl avatar matthew-brett avatar nonzero avatar radarhere avatar tomkooij avatar

Stargazers

 avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

pytables-wheels's Issues

AVX(2) instructions in manylinux i686 wheels

I'm experiencing Illegal instruction exceptions on manylinux i686 wheels on several (recent) CPUs:

  • Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
  • Intel(R) Celeron(R) CPU N2830 @ 2.16GHz
    Illegal instructions also reported in PyTables/PyTables#626

Playing with objdump I'm seeing AVX instructions in libhdf5extension.so in py27 manylinux wheel for v3.4.2. From the travis log it seems pytables is build using -mavx2 which should not happen for an i686 build.

@matthew-brett: Can you help setting the correct compiler arch -m flags?
(This might be due to the fact we are now building internal blosc, instead of the separate blosc lib)

travis log: (note the -mavx2 flag)

building 'tables.hdf5extension' extension
gcc -pthread -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -std=gnu99 -fPIC -DNDEBUG=1 -DHAVE_LZO2_LIB=1 -DHAVE_BZ2_LIB=1 -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -Ihdf5-blosc/src -Ic-blosc/blosc -Ic-blosc/internal-complibs/zlib-1.2.8 -Ic-blosc/internal-complibs/lz4-1.7.5 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zstd-1.1.3 -Ic-blosc/internal-complibs/zstd-1.1.3/common -Ic-blosc/internal-complibs/zstd-1.1.3 -I/opt/python/cp27-cp27mu/lib/python2.7/site-packages/numpy/core/include -I/opt/python/cp27-cp27mu/include/python2.7 -c tables/hdf5extension.c -o build/temp.linux-i686-2.7/tables/hdf5extension.o -std=gnu99 -Isrc -DH5_USE_18_API -DH5Acreate_vers=2 -DH5Aiterate_vers=2 -DH5Dcreate_vers=2 -DH5Dopen_vers=2 -DH5Eclear_vers=2 -DH5Eprint_vers=2 -DH5Epush_vers=2 -DH5Eset_auto_vers=2 -DH5Eget_auto_vers=2 -DH5Ewalk_vers=2 -DH5E_auto_t_vers=2 -DH5Gcreate_vers=2 -DH5Gopen_vers=2 -DH5Pget_filter_vers=2 -DH5Pget_filter_by_id_vers=2 -DH5Tarray_create_vers=2 -DH5Tget_array_dims_vers=2 -DH5Z_class_t_vers=2 -DSHUFFLE_SSE2_ENABLED -msse2 -DSHUFFLE_AVX2_ENABLED -mavx2

Update multibuild to a version that strips binaries in wheels

On the next release, please update the multibuild submodule to matthew-brett/multibuild@b2748e5 or newer. This should reduce the size of the pytables wheels from 4.2M to 2.6M by excluding unnecessary debug symbols. The more important change is that the extracted files on disk are significantly smaller, e.g. when you install pytables in a docker container, then the size reduction is very noticeable.

Archive this repo?

Can we archive this repo? Many of the multibuild projects under this org have moved to cibuildwheel, and now build via CI jobs on the main repo. Is this true of pytables?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.