Package publishing issues on Ubuntu 16.04 about oasislmf HOT 20 CLOSED

We may not need the wheel and instead should just do the source dist

from oasislmf.

So wheels are used to deploy the compiled python and extensions but since we need to pull ktools on deploy I don't think we can make one.

from oasislmf.

Okay, so can we just drop the code check for the wheel package then?

from oasislmf.

yeah i think so

from oasislmf.

Ok, ive removed it, ill try deploying now

from oasislmf.

The install worked for me, let me know if you have any other issues, you may need to run the install with -v to get the logs.

from oasislmf.

Okay, it did install this time though took a little while. It seems to throw a few errors during the install, though these could be ktools issues.

from oasislmf.

It also seemed to run the compilation tests etc twice. Why would that be?

from oasislmf.

I'm not sure about it running twice, I'm not seeing it. We run ./autogen.sh && ./configure && make && make check

from oasislmf.

See attached - it downloads and build ktools at lines 4139 and 7443. This is the output of pip install -v oasislmf.

build.out.txt

from oasislmf.

That's very strange, What versions of python/pip are you using? It may be that it fails on the first build and so retries.

from oasislmf.

I'm using the following Dockerfile:

FROM ubuntu:16.04

RUN apt update
RUN apt install -y python-setuptools python-pip g++ build-essential libtool zlib1g-dev autoconf unixodbc-dev
RUN pip install -v oasislmf

ENTRYPOINT bash

The versions are:
root@c2738e9ae3b9:/# python --version
Python 2.7.12
root@c2738e9ae3b9:/# pip --version
pip 8.1.1 from /usr/lib/python2.7/dist-packages (python 2.7)

from oasislmf.

Also, @benhayes21 has had similar issues with the install seeming to hang intermittently.

from oasislmf.

I have seen the hang once, it seems there is sometimes an issue getting the ktools tar. I suspect we need to just add a timeout and retry a couple of times before failing if we cant retrieve it.

from oasislmf.

I've just published the package with a timeout on the tar fetching

from oasislmf.

Ok, after some digging it seems the reason I wasn't seeing it is that wheel doesn't get installed into the venv by default but it does when you install through apt-get.

We don't have a lot of control over the pip install process which will try to build the wheel if wheel is installed but if the command to build the wheel fails it will fall back to sdist which is what we want.

I have added a dummy command for sdist_wheel which just exits with an error code and this seems to solve the issue although there is now an error message saying that it failed to build the wheel.

from oasislmf.

There were no wheel issues on my Mac with version 0.0.1, for which I did add a universal wheel. I think pip detected the wheel and installed it into the virtual env. Perhaps apt works differently.

For 0.0.5 I've noted a wheel error during the pip install.

  Building wheels for collected packages: oasislmf
      Running setup.py bdist_wheel for oasislmf ... error
      Complete output from command /Users/srm/Documents/sandeep/cst/dev/oasis/OasisLMF/oasislmf-venv/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/vx/3ts13s190k50b2gzt_6zyp0m0000gn/T/pip-build-8eC0n5/oasislmf/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /var/folders/vx/3ts13s190k50b2gzt_6zyp0m0000gn/T/tmpMiEBB1pip-wheel- --python-tag cp27:
      running bdist_wheel

      ----------------------------------------
      Failed building wheel for oasislmf

But the install worked so clearly it has installed it from the source dist.

I think it is a good idea to have a universal wheel in addition to a source dist. I think it would work for most people, and it is a faster process.

from oasislmf.

There were no issues on 0.0.1 because ktools wasn't included.

We cant have a universal wheel unless we create binaries for ktools for each platform. This is why we force the wheel build to fail so that it reverts to src install.

I agree a universal wheel would be a good idea but as far as i can see its not possible without either building ktools for all environments or adding ktools as extensions which would require some refactoring as it would be too inefficient.

from oasislmf.

Having the binaries pre-packaged might be the better option, if it gives a faster and more reliable install process. @bmatharu any thoughts? What permutation would we need to build?

from oasislmf.

Prebuild binaries is probably the easiest approach.
We will need one set for each platform supported - suggest statically linked to stop any shared library issues.
Need builds for
Linux
Windows
OS X??
cygwin??

I assume we're only going to support 64bit environments

from oasislmf.