peterjc / backports.lzma Goto Github PK

View Code? Open in Web Editor NEW

56.0 56.0 30.0 214 KB

Backport of Python 3.3's standard library module lzma for LZMA/XY compressed files

License: BSD 3-Clause "New" or "Revised" License

Python 69.62% C 30.38%

backports.lzma's People

Contributors

Stargazers

Watchers

backports.lzma's Issues

Inefficient buffer growth algorithm

Please read https://bugs.python.org/issue41265

backports.lzma has the same problem, the code:

    #if BUFSIZ < 8192
    #define INITIAL_BUFFER_SIZE 8192
    #else
    #define INITIAL_BUFFER_SIZE BUFSIZ
    #endif

    static int
    grow_buffer(PyObject **buf)
    {
        size_t size = PyBytes_GET_SIZE(*buf);
        return _PyBytes_Resize(buf, size + (size >> 3) + 6);
    }

backports.lzma can use a new growth algorithm, and switch to CPython's algorithm after it changes.

Failed to download backports.lzma v0.0.4

From docker-archive/docker-registry#572:

Downloading/unpacking backports.lzma>=0.0.2
  Downloading backports.lzma-0.0.4.tar.gz
  Running setup.py (path:/tmp/pip_build_root/backports.lzma/setup.py) egg_info for package backports.lzma
    This is backports.lzma version 0.0.4

    error: Namespace package problem: backports is a namespace package, but its
    __init__.py does not call declare_namespace()! Please fix it.
    (See the setuptools manual under "Namespace Packages" for details.)
    "
    Complete output from command python setup.py egg_info:
    This is backports.lzma version 0.0.4

xz compressed pickled pandas DataFrame does not finish decompressing

Python 2.7.17, installed via pyenv on macOS 10.14.6 with xz via homebrew.

A ~10MB xz compressed (~130MB decompressed) pandas DataFrame does not decompress after waiting at least a minute, clearly in a loop or slow as the machine fans are running at high speed. htop does not show very much CPU or memory use however.

$ python --version

Python 2.7.17

$ pip list
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package         Version
--------------- -------
backports.lzma  0.0.14
numpy           1.16.6
pandas          0.19.2
pip             20.0.2
python-dateutil 2.8.1
pytz            2019.3
setuptools      41.2.0
six             1.14.0

$ python - << "EOP"
import pickle, backports.lzma
with backports.lzma.open('guideseq_data.pkl.xz') as m:
    pickle.load(m)
EOP

^CTraceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "/Users/testuser/.pyenv/versions/2.7.17/lib/python2.7/pickle.py", line 1384, in load
    return Unpickler(file).load()
  File "/Users/testuser/.pyenv/versions/2.7.17/lib/python2.7/pickle.py", line 863, in load
    key = read(1)
  File "/Users/testuser/.pyenv/versions/2.7.17/lib/python2.7/site-packages/backports/lzma/__init__.py", line 295, in read
    return self._read_block(size)
  File "/Users/testuser/.pyenv/versions/2.7.17/lib/python2.7/site-packages/backports/lzma/__init__.py", line 257, in _read_block
    self._buffer = self._buffer[n:]
KeyboardInterrupt

setup.py: namespace_packages is setuptools option (while using distutils)

The setup.py script uses plain distutils. However, it passes:

  namespace_packages = ['backports'],

This is an option specific to setuptools and it does not do anything in distutils, as setup.py warns:

/usr/lib64/python3.5/distutils/dist.py:261: UserWarning: Unknown distribution option: 'namespace_packages'
  warnings.warn(msg)

Error for Running setup.py

I am using python 3.7.3 and pip3 on my Mac OS. The wrong message shows:

$ pip3 install backports.lzma

Collecting backports.lzma
  Downloading backports.lzma-0.0.14.tar.gz (47 kB)
     |████████████████████████████████| 47 kB 287 kB/s 
Using legacy setup.py install for backports.lzma, since package 'wheel' is not installed.
Installing collected packages: backports.lzma
    Running setup.py install for backports.lzma ... error
    ERROR: Command errored out with exit status 1:
     command: /Users/oscar/.pyenv/versions/3.7.3/bin/python3.7 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/jk/xqfdqmw93ygbzcqdls0rkz9c0000gn/T/pip-install-md9jdec6/backports.lzma/setup.py'"'"'; __file__='"'"'/private/var/folders/jk/xqfdqmw93ygbzcqdls0rkz9c0000gn/T/pip-install-md9jdec6/backports.lzma/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/jk/xqfdqmw93ygbzcqdls0rkz9c0000gn/T/pip-record-es8vj5l9/install-record.txt --single-version-externally-managed --compile --install-headers /Users/oscar/.pyenv/versions/3.7.3/include/python3.7m/backports.lzma
         cwd: /private/var/folders/jk/xqfdqmw93ygbzcqdls0rkz9c0000gn/T/pip-install-md9jdec6/backports.lzma/
    Complete output (21 lines):
    This is backports.lzma version 0.0.14
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-10.15-x86_64-3.7
    creating build/lib.macosx-10.15-x86_64-3.7/backports
    copying backports/__init__.py -> build/lib.macosx-10.15-x86_64-3.7/backports
    creating build/lib.macosx-10.15-x86_64-3.7/backports/lzma
    copying backports/lzma/__init__.py -> build/lib.macosx-10.15-x86_64-3.7/backports/lzma
    running build_ext
    building 'backports.lzma._lzma' extension
    creating build/temp.macosx-10.15-x86_64-3.7
    creating build/temp.macosx-10.15-x86_64-3.7/backports
    creating build/temp.macosx-10.15-x86_64-3.7/backports/lzma
    clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Users/oscar/.pyenv/versions/3.7.3/include -I/Users/oscar/include -I/opt/local/include -I/usr/local/include -I/Users/oscar/.pyenv/versions/3.7.3/include/python3.7m -c backports/lzma/_lzmamodule.c -o build/temp.macosx-10.15-x86_64-3.7/backports/lzma/_lzmamodule.o
    backports/lzma/_lzmamodule.c:115:10: fatal error: 'lzma.h' file not found
    #include <lzma.h>
             ^~~~~~~~
    1 error generated.
    error: command 'clang' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/oscar/.pyenv/versions/3.7.3/bin/python3.7 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/jk/xqfdqmw93ygbzcqdls0rkz9c0000gn/T/pip-install-md9jdec6/backports.lzma/setup.py'"'"'; __file__='"'"'/private/var/folders/jk/xqfdqmw93ygbzcqdls0rkz9c0000gn/T/pip-install-md9jdec6/backports.lzma/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/jk/xqfdqmw93ygbzcqdls0rkz9c0000gn/T/pip-record-es8vj5l9/install-record.txt --single-version-externally-managed --compile --install-headers /Users/oscar/.pyenv/versions/3.7.3/include/python3.7m/backports.lzma Check the logs for full command output.

Any suggestions are appreciated!

I cannot install backports.lzma via pip on my macOS HighSierra 10.13.1

It worked well on the previous version of macOS.

Now, the error log is as follows:

pip install backports.lzma
Collecting backports.lzma
  Using cached backports.lzma-0.0.8.tar.gz
Building wheels for collected packages: backports.lzma
  Running setup.py bdist_wheel for backports.lzma ... error
  Complete output from command /anaconda2/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/db/9r33p7t972dfrhkgw2h549sw0000gn/T/pip-build-M90aY7/backports.lzma/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /var/folders/db/9r33p7t972dfrhkgw2h549sw0000gn/T/tmpxhVLp6pip-wheel- --python-tag cp27:
  This is backports.lzma version 0.0.8
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-10.6-x86_64-2.7
  creating build/lib.macosx-10.6-x86_64-2.7/backports
  copying backports/__init__.py -> build/lib.macosx-10.6-x86_64-2.7/backports
  creating build/lib.macosx-10.6-x86_64-2.7/backports/lzma
  copying backports/lzma/__init__.py -> build/lib.macosx-10.6-x86_64-2.7/backports/lzma
  running build_ext
  building 'backports/lzma/_lzma' extension
  creating build/temp.macosx-10.6-x86_64-2.7
  creating build/temp.macosx-10.6-x86_64-2.7/backports
  creating build/temp.macosx-10.6-x86_64-2.7/backports/lzma
  gcc -fno-strict-aliasing -I/anaconda2/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/anaconda2/include -I/Users/tarkhov/include -I/opt/local/include -I/usr/local/include -I/anaconda2/include/python2.7 -c backports/lzma/_lzmamodule.c -o build/temp.macosx-10.6-x86_64-2.7/backports/lzma/_lzmamodule.o
  xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun
  error: command 'gcc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for backports.lzma
  Running setup.py clean for backports.lzma
Failed to build backports.lzma
Installing collected packages: backports.lzma
  Running setup.py install for backports.lzma ... error
    Complete output from command /anaconda2/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/db/9r33p7t972dfrhkgw2h549sw0000gn/T/pip-build-M90aY7/backports.lzma/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/folders/db/9r33p7t972dfrhkgw2h549sw0000gn/T/pip-aYS63q-record/install-record.txt --single-version-externally-managed --compile:
    This is backports.lzma version 0.0.8
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-10.6-x86_64-2.7
    creating build/lib.macosx-10.6-x86_64-2.7/backports
    copying backports/__init__.py -> build/lib.macosx-10.6-x86_64-2.7/backports
    creating build/lib.macosx-10.6-x86_64-2.7/backports/lzma
    copying backports/lzma/__init__.py -> build/lib.macosx-10.6-x86_64-2.7/backports/lzma
    running build_ext
    building 'backports/lzma/_lzma' extension
    creating build/temp.macosx-10.6-x86_64-2.7
    creating build/temp.macosx-10.6-x86_64-2.7/backports
    creating build/temp.macosx-10.6-x86_64-2.7/backports/lzma
    gcc -fno-strict-aliasing -I/anaconda2/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/anaconda2/include -I/Users/tarkhov/include -I/opt/local/include -I/usr/local/include -I/anaconda2/include/python2.7 -c backports/lzma/_lzmamodule.c -o build/temp.macosx-10.6-x86_64-2.7/backports/lzma/_lzmamodule.o
    xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun
    error: command 'gcc' failed with exit status 1

    ----------------------------------------
Command "/anaconda2/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/db/9r33p7t972dfrhkgw2h549sw0000gn/T/pip-build-M90aY7/backports.lzma/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/folders/db/9r33p7t972dfrhkgw2h549sw0000gn/T/pip-aYS63q-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/db/9r33p7t972dfrhkgw2h549sw0000gn/T/pip-build-M90aY7/backports.lzma/

Splitting .py from .so

I've been packaging this for openSUSE, and needed to manually split the package into two parts, one for sitelib and the other for sitearch.

The primary problem is that if PyPi package backports, or any other backports.foo module is installed, there is reasonable chance that they are installed into the arch-independent site-packages area, while this package will be installed into the arch-dependent site-packages. And Python 2 doesnt handle the same package appear in both, so the top level backports appearing in both is problematic. Many installs of Python have one area for both - the split is more typical in distro installs, where separating arch from non-arch files is important.

To solve this, I manually move the backports/lzma/init.py into sitelib , and move the _lzma.so out of the backports namespace, to the top level of sitearch.

You can see the .spec at https://build.opensuse.org/package/show/home:jayvdb:py-backports/python-backports.lzma , and the build logs are also there if you login to OBS.

I believe (but not 100% sure) this can be fixed in setup.py, and I am happy to do that, if the maintainers are happy to have the dso moved to the top level. There is likely some extra voodoo needed to tell setuptools that the package is using two top level namespaces.

If not, it is a fairly standard type of 'packaging' problem that doesnt need to be fixed here.

lzma.LZMAFile: TemporaryFile instance has no attribute 'seekable'

Long story short, seekable() is not a part of official file object API in Python 2.7. In particular, TemporaryFile and subclasses don't have that, and therefore LZMAFile fails on them. You shouldn't rely on undocumented methods for the backport.

Reference: https://docs.python.org/2/library/stdtypes.html#file-objects

test_lzma.py ImportError: cannot import name lzma

On Debian 10, python 2.7 I get this error after installing backports.lzma

Traceback (most recent call last): File "test_lzma.py", line 35, in <module> from backports import lzma ImportError: cannot import name lzma

Any idea what is wrong?
Installed Ubuntu 18 LTS and it seems to be working

Tests fail when liblzma is built without uncommon filters

In Gentoo we support a minimal build of xz-utils that includes only the commonly used filters (i.e. CRC32+64, LZMA1+2 and relevant match filters). Sadly, it seems that the tests rely on some less common filters being present and fail in this configuration:

======================================================================
ERROR: test_decompressor_raw_2 (__main__.CompressorDecompressorTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_lzma.py", line 142, in test_decompressor_raw_2
    lzd = LZMADecompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_2)
LZMAError: Invalid or unsupported options

======================================================================
ERROR: test_decompressor_raw_3 (__main__.CompressorDecompressorTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_lzma.py", line 146, in test_decompressor_raw_3
    lzd = LZMADecompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_3)
LZMAError: Invalid or unsupported options

======================================================================
ERROR: test_decompressor_raw_4 (__main__.CompressorDecompressorTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_lzma.py", line 150, in test_decompressor_raw_4
    lzd = LZMADecompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_4)
LZMAError: Invalid or unsupported options

======================================================================
ERROR: test_roundtrip_raw (__main__.CompressorDecompressorTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_lzma.py", line 199, in test_roundtrip_raw
    lzc = LZMACompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_4)
LZMAError: Invalid or unsupported options

======================================================================
ERROR: test_decompress_good_input (__main__.CompressDecompressFunctionTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_lzma.py", line 313, in test_decompress_good_input
    COMPRESSED_RAW_2, lzma.FORMAT_RAW, filters=FILTERS_RAW_2)
  File "/tmp/portage/dev-python/backports-lzma-0.0.10/work/backports.lzma-0.0.10-pypy/lib/backports/lzma/__init__.py", line 463, in decompress
    decomp = LZMADecompressor(format, memlimit, filters)
LZMAError: Invalid or unsupported options

======================================================================
ERROR: test_roundtrip (__main__.CompressDecompressFunctionTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_lzma.py", line 364, in test_roundtrip
    cdata = lzma.compress(INPUT, lzma.FORMAT_RAW, filters=FILTERS_RAW_4)
  File "/tmp/portage/dev-python/backports-lzma-0.0.10/work/backports.lzma-0.0.10-pypy/lib/backports/lzma/__init__.py", line 449, in compress
    comp = LZMACompressor(format, check, preset, filters)
LZMAError: Invalid or unsupported options

======================================================================
ERROR: test_iterator (__main__.FileTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_lzma.py", line 823, in test_iterator
    format=lzma.FORMAT_RAW, filters=FILTERS_RAW_2) as f:
  File "/tmp/portage/dev-python/backports-lzma-0.0.10/work/backports.lzma-0.0.10-pypy/lib/backports/lzma/__init__.py", line 113, in __init__
    self._decompressor = LZMADecompressor(**self._init_args)
LZMAError: Invalid or unsupported options

======================================================================
ERROR: test_read (__main__.FileTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_lzma.py", line 655, in test_read
    format=lzma.FORMAT_RAW, filters=FILTERS_RAW_2) as f:
  File "/tmp/portage/dev-python/backports-lzma-0.0.10/work/backports.lzma-0.0.10-pypy/lib/backports/lzma/__init__.py", line 113, in __init__
    self._decompressor = LZMADecompressor(**self._init_args)
LZMAError: Invalid or unsupported options

======================================================================
ERROR: test_read_multistream (__main__.FileTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_lzma.py", line 694, in test_read_multistream
    format=lzma.FORMAT_RAW, filters=FILTERS_RAW_3) as f:
  File "/tmp/portage/dev-python/backports-lzma-0.0.10/work/backports.lzma-0.0.10-pypy/lib/backports/lzma/__init__.py", line 113, in __init__
    self._decompressor = LZMADecompressor(**self._init_args)
LZMAError: Invalid or unsupported options

======================================================================
ERROR: test_write (__main__.FileTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_lzma.py", line 857, in test_write
    filters=FILTERS_RAW_2) as f:
  File "/tmp/portage/dev-python/backports-lzma-0.0.10/work/backports.lzma-0.0.10-pypy/lib/backports/lzma/__init__.py", line 120, in __init__
    preset=preset, filters=filters)
LZMAError: Invalid or unsupported options

----------------------------------------------------------------------

I think we should look into more graceful handling of missing liblzma features.

Use macOS's included liblzma?

Cross reference samtools/htslib#628

Apparently a vanilla macOS installation in fact includes /usr/lib/liblzma.dylib but sadly does not ship the corresponding header files. We could in principle include a fake lzma_stub.h header that provides just enough declarations so that the Python lzma wrapper can be compiled, for use if we have a usable liblzma but no lzma.h. This may require trickery in our setup.py to achieve this...

backports/lzma/_lzmamodule.c(115) : fatal error C1083: Cannot open include file: 'lzma.h': No such file or directory

I have a Windows 7 PC with Visual Studio 2015 upgrade 3 installed on it and I would like to install backports.lzma library for Python with pip.

I have downloaded and extracted XZ Utils in Program Files folder. (XZ Utils is mandatory for install backports library as specified from Pypi site )

This is the pip output:

Installing collected packages: backports.lzma
Running setup.py install for backports.lzma: started
Running setup.py install for backports.lzma: finished with status 'error'
Complete output from command "C:\Program Files (x86)\Python27\python.exe" -u -c "import setuptools, tokenize;file='c:\users\myuser\appdata\local\temp\pip-build-smbn4k\backports.lzma\setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record c:\users\myuser\appdata\local\temp\pip-hfgg8w-record\install-record.txt --single-version-externally-managed --compile:
This is backports.lzma version 0.0.8
running install
running build
running build_py
creating build
creating build\lib.win32-2.7
creating build\lib.win32-2.7\backports
copying backports_init_.py -> build\lib.win32-2.7\backports
creating build\lib.win32-2.7\backports\lzma
copying backports\lzma_init_.py -> build\lib.win32-2.7\backports\lzma
running build_ext
building 'backports/lzma/_lzma' extension
creating build\temp.win32-2.7
creating build\temp.win32-2.7\Release
creating build\temp.win32-2.7\Release\backports
creating build\temp.win32-2.7\Release\backports\lzma
C:\Users\myuser\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG "-IC:\Program Files (x86)\Python27\include" -IC:\Users\myuser\include -I/opt/local/include -I/usr/local/include "-IC:\Program Files (x86)\Python27\include" "-IC:\Program Files (x86)\Python27\PC" /Tcbackports/lzma/_lzmamodule.c /Fobuild\temp.win32-2.7\Release\backports/lzma/_lzmamodule.obj
_lzmamodule.c
backports/lzma/_lzmamodule.c(32) : warning C4273: 'PyErr_NewExceptionWithDoc' : inconsistent dll linkage
C:\Program Files (x86)\Python27\include\pyerrors.h(226) : see previous definition of 'PyErr_NewExceptionWithDoc'
backports/lzma/_lzmamodule.c(115) : fatal error C1083: Cannot open include file: 'lzma.h': No such file or directory
error: command 'C:\Users\myuser\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\cl.exe' failed with exit status 2

Failed building wheel for backports.lzma
Command ""C:\Program Files (x86)\Python27\python.exe" -u -c "import setuptools, tokenize;file='c:\users\myuser\appdata\local\temp\pip-build-smbn4k\backports.lzma\setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record c:\users\myuser\appdata\local\temp\pip-hfgg8w-record\install-record.txt --single-version-externally-managed --compile" failed with error code 1 in c:\users\myuser\appdata\local\temp\pip-build-smbn4k\backports.lzma
'backports.lzma' failed to install. Exit code: 1

However in Program Files\xz\include there is the lzma.h file.
How to link the header to be recognized?

lzma sometimes fails to decompress a file

Bacports.lzma fails to decompress some files, even though a direct call to XZ Utils will process the same files without complaints.

System details:
OS X 10.9.3
Python 2.7.7 via Anaconda 2.0.0
backports.lzma 0.0.2 (Just noticed there is a new version, but couldn't be bothered with the installation if it is only for the unicode support.)

Unfortunately I don't have Py3 installed and cannot therefore tell if this is an issue of the backported or the original version. I'm putting the issue here in order to get started somewhere.

Example data files:
This one fails:
https://dl.dropboxusercontent.com/u/90169773/lzma_issue/22h_ticks_bad.bi5
This one is processed without errors:
https://dl.dropboxusercontent.com/u/90169773/lzma_issue/23h_ticks_good.bi5

Attempting to decompress the 'bad' file raises the following error:
EOFError: Compressed file ended before the end-of-stream marker was reached.

The example files contain tick data and have been downloaded from the Dukascopy(.com) bank's historical economic data service. This error is relatively rare: Typically there are 1-5 failures per each 1000 data files. So far, all of them have been recoverable with XZ Utils. I have not detected any obvious pattern in which ones fail and which ones won't.

Heres a snippet of my program, including the xz workaround:

from future import print_function, division, absolute_import, unicode_literals
import os.path as op
import backports.lzma as lz
import subprocess

class whatever():

@staticmethod
def decompress(path):
    # Got data at all?
    if not op.isfile(path):
        raise Exception('Cannot find the .bi5 file!')
    try:
        # Regular decompression, ok most of the time
        with lz.open(path.encode('utf-8')) as f:
            filecontent = f.read()
    except EOFError:
        # back-up decompression via calling XZ Utils
        xz = '/usr/local/bin/xz -d --format=lzma --keep --suffix=.bi5 ' + '"' + path + '"'
        print('LZMA failed, falling back to XZ Utils for', path)
        __ = subprocess.call(xz.decode('utf-8'), shell=True)
        # The file should now be decompressed on disk. Strip the suffix and read it.
        with open(path[:-4], mode='r') as f:
            filecontent = f.read()
    return filecontent

pypy3: implicit declaration of function ‘PyUnicode_DecodeUTF8Stateful’

Now that we know how to solve PyPy problems, time to look at PyPy3 ;-). Not that it's very important given that it has a builtin lzma module but still we could fix it for completeness. The error is:

cc -pthread -fPIC -I/usr/lib64/pypy3/include -I/home/mgorny/include -I/opt/local/include -I/usr/local/include -I/usr/lib64/pypy3/include -c backports/lzma/_lzmamodule.c -o build/temp.linux-x86_64-3.5/backports/lzma/_lzmamodule.o
backports/lzma/_lzmamodule.c: In function ‘_PyUnicode_FromId’:
backports/lzma/_lzmamodule.c:87:18: warning: implicit declaration of function ‘PyUnicode_DecodeUTF8Stateful’ [-Wimplicit-function-declaration]
     id->object = PyUnicode_DecodeUTF8Stateful(id->string,
                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
backports/lzma/_lzmamodule.c:87:16: warning: assignment makes pointer from integer without a cast [-Wint-conversion]
     id->object = PyUnicode_DecodeUTF8Stateful(id->string,
                ^

My initial guess is that the PyUnicode_DecodeUTF8Stateful is an internal CPython function that extensions are not supposed to use. Documentation does not include that function, and random stackoverflow post seems to confirm that.

Failed to install the latest backports.lzma - 0.0.14

as you can see below:

$ pip install backports.lzma
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting backports.lzma
  Using cached https://files.pythonhosted.org/packages/21/0f/1a9990233076d48aa2084100ba289ca162975e73a688f3a56c0ee2bb441a/backports.lzma-0.0.14.tar.gz
Building wheels for collected packages: backports.lzma
  Building wheel for backports.lzma (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /var/lib/jenkins/workspace/test-job/.env/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-MeXJYh/backports.lzma/setup.py'"'"'; __file__='"'"'/tmp/pip-install-MeXJYh/backports.lzma/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-PMNMSq --python-tag cp27
       cwd: /tmp/pip-install-MeXJYh/backports.lzma/
  Complete output (21 lines):
  This is backports.lzma version 0.0.14
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-2.7
  creating build/lib.linux-x86_64-2.7/backports
  copying backports/__init__.py -> build/lib.linux-x86_64-2.7/backports
  creating build/lib.linux-x86_64-2.7/backports/lzma
  copying backports/lzma/__init__.py -> build/lib.linux-x86_64-2.7/backports/lzma
  running build_ext
  building 'backports.lzma._lzma' extension
  creating build/temp.linux-x86_64-2.7
  creating build/temp.linux-x86_64-2.7/backports
  creating build/temp.linux-x86_64-2.7/backports/lzma
  gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/var/lib/jenkins/workspace/test-job/.env/include -I/root/include -I/opt/local/include -I/usr/local/include -I/usr/include/python2.7 -c backports/lzma/_lzmamodule.c -o build/temp.linux-x86_64-2.7/backports/lzma/_lzmamodule.o
  backports/lzma/_lzmamodule.c:115:18: fatal error: lzma.h: No such file or directory
   #include <lzma.h>
                    ^
  compilation terminated.
  error: command 'gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for backports.lzma
  Running setup.py clean for backports.lzma
Failed to build backports.lzma
Installing collected packages: backports.lzma
  Running setup.py install for backports.lzma ... error
    ERROR: Command errored out with exit status 1:
     command: /var/lib/jenkins/workspace/test-job/.env/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-MeXJYh/backports.lzma/setup.py'"'"'; __file__='"'"'/tmp/pip-install-MeXJYh/backports.lzma/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-AVFFt0/install-record.txt --single-version-externally-managed --compile --install-headers /var/lib/jenkins/workspace/test-job/.env/include/site/python2.7/backports.lzma
         cwd: /tmp/pip-install-MeXJYh/backports.lzma/
    Complete output (21 lines):
    This is backports.lzma version 0.0.14
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-2.7
    creating build/lib.linux-x86_64-2.7/backports
    copying backports/__init__.py -> build/lib.linux-x86_64-2.7/backports
    creating build/lib.linux-x86_64-2.7/backports/lzma
    copying backports/lzma/__init__.py -> build/lib.linux-x86_64-2.7/backports/lzma
    running build_ext
    building 'backports.lzma._lzma' extension
    creating build/temp.linux-x86_64-2.7
    creating build/temp.linux-x86_64-2.7/backports
    creating build/temp.linux-x86_64-2.7/backports/lzma
    gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/var/lib/jenkins/workspace/test-job/.env/include -I/root/include -I/opt/local/include -I/usr/local/include -I/usr/include/python2.7 -c backports/lzma/_lzmamodule.c -o build/temp.linux-x86_64-2.7/backports/lzma/_lzmamodule.o
    backports/lzma/_lzmamodule.c:115:18: fatal error: lzma.h: No such file or directory
     #include <lzma.h>
                      ^
    compilation terminated.
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /var/lib/jenkins/workspace/test-job/.env/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-MeXJYh/backports.lzma/setup.py'"'"'; __file__='"'"'/tmp/pip-install-MeXJYh/backports.lzma/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-AVFFt0/install-record.txt --single-version-externally-managed --compile --install-headers /var/lib/jenkins/workspace/test-job/.env/include/site/python2.7/backports.lzma Check the logs for full command output.

PyPy support

PyPy was (?) working according to discussion on #2, but appears to be broken again, e.g.

$ pypy2.3 test_lzma.py
...
test_simple_bad_args (__main__.CompressorDecompressorTestCase) ... ok
test_bad_args (__main__.CompressDecompressFunctionTestCase) ... ERROR
test_decompress_bad_input (__main__.CompressDecompressFunctionTestCase) ... ok
...

======================================================================
ERROR: test_bad_args (__main__.CompressDecompressFunctionTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_lzma.py", line 269, in test_bad_args
    b"", format=lzma.FORMAT_RAW, filters={})
  File "/Users/pjcock/Downloads/Software/pypy-2.3-osx64/lib-python/2.7/unittest/case.py", line 475, in assertRaises
    callableObj(*args, **kwargs)
  File "/Users/pjcock/Downloads/Software/pypy-2.3-osx64/site-packages/backports/lzma/__init__.py", line 457, in decompress
    decomp = LZMADecompressor(format, memlimit, filters)
LZMAError: Internal error

----------------------------------------------------------------------
Ran 94 tests in 7.147s

FAILED (errors=1)
Traceback (most recent call last):
  File "app_main.py", line 75, in run_toplevel
  File "test/test_lzma.py", line 1571, in <module>
    test_main()
  File "test/test_lzma.py", line 1567, in test_main
    MiscellaneousTestCase,
  File "/Users/pjcock/Downloads/Software/pypy-2.3-osx64/lib-python/2.7/test/test_support.py", line 1261, in run_unittest
    _run_suite(suite)
  File "/Users/pjcock/Downloads/Software/pypy-2.3-osx64/lib-python/2.7/test/test_support.py", line 1215, in _run_suite
    raise TestFailed(err)
TestFailed: Traceback (most recent call last):
  File "test/test_lzma.py", line 269, in test_bad_args
    b"", format=lzma.FORMAT_RAW, filters={})
  File "/Users/pjcock/Downloads/Software/pypy-2.3-osx64/lib-python/2.7/unittest/case.py", line 475, in assertRaises
    callableObj(*args, **kwargs)
  File "/Users/pjcock/Downloads/Software/pypy-2.3-osx64/site-packages/backports/lzma/__init__.py", line 457, in decompress
    decomp = LZMADecompressor(format, memlimit, filters)
LZMAError: Internal error

0.0.5 Broken on PyPI

Looks like the recently pushed out release is broken the same way 0.0.4 was broken. Below is the code to reproduce via a clean docker image of python 2

$ docker run -it python bash
root@7d7e1c19a61d:/# pip install backports.lzma
Collecting backports.lzma
  Downloading backports.lzma-0.0.5.tar.gz
    Complete output from command python setup.py egg_info:
    This is backports.lzma version 0.0.5
    running egg_info
    creating pip-egg-info/backports.lzma.egg-info
    writing namespace_packages to pip-egg-info/backports.lzma.egg-info/namespace_packages.txt
    writing top-level names to pip-egg-info/backports.lzma.egg-info/top_level.txt
    writing dependency_links to pip-egg-info/backports.lzma.egg-info/dependency_links.txt
    writing pip-egg-info/backports.lzma.egg-info/PKG-INFO
    writing manifest file 'pip-egg-info/backports.lzma.egg-info/SOURCES.txt'
    warning: manifest_maker: standard file '-c' not found

    error: Namespace package problem: backports is a namespace package, but its
    __init__.py does not call declare_namespace()! Please fix it.
    (See the setuptools manual under "Namespace Packages" for details.)
    "

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-jq8ug_ce/backports.lzma/
root@7d7e1c19a61d:/# pip install backports.lzma=0.0.3
Invalid requirement: 'backports.lzma=0.0.3'
= is not a valid operator. Did you mean == ?

root@7d7e1c19a61d:/# pip install backports.lzma==0.0.3
Collecting backports.lzma==0.0.3
  Downloading backports.lzma-0.0.3.tar.gz
Installing collected packages: backports.lzma
  Running setup.py install for backports.lzma ... done
Successfully installed backports.lzma-0.0.3

Windows wheels available

In case of interest: windows wheels for (py-2.6 and py-2.7) of backports.lzma are provided at:
https://anaconda.org/carlkl/backports.lzma

install it with:

pip install -i https://pypi.anaconda.org/carlkl/simple backports.lzma

setup.py broken on PyPy

The most recent version seems to be broken on PyPy:

Collecting backports.lzma
  Using cached backports.lzma-0.0.9.tar.gz
    Complete output from command python setup.py egg_info:
    This is backports.lzma version 0.0.9
    Traceback (most recent call last):
      File "<module>", line 1, in <module>
      File "/tmp/pip-build-OiRNki/backports.lzma/setup.py", line 32, in <module>
        is32bit = tuple.__itemsize__ == 4
    AttributeError: type object 'tuple' has no attribute '__itemsize__'
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-OiRNki/backports.lzma/

You seem to be accessing deep internals of CPython which looks like a very, very bad idea.

'setup.py build' fails with LLD linker on a lib64 system

The setup.py is wrongly forcing /usr/lib path on amd64 systems:

                    library_dirs = [
                        os.path.join(prefix, 'lib'),
                        os.path.join(home, 'lib'),
                        '/opt/local/lib',
                        '/usr/local/lib'
                    ]

GNU ld outputs warnings because of it:

$ python setup.py build
This is backports.lzma version 0.0.14
running build
running build_py
running build_ext
building 'backports.lzma._lzma' extension
creating build/temp.linux-x86_64-3.9
creating build/temp.linux-x86_64-3.9/backports
creating build/temp.linux-x86_64-3.9/backports/lzma
x86_64-pc-linux-gnu-gcc-9.3.0 -pthread -fPIC -I/usr/lib/python-exec/python3.9/../../../include -I/home/mgorny/include -I/opt/local/include -I/usr/local/include -I/usr/include/python3.9 -c backports/lzma/_lzmamodule.c -o build/temp.linux-x86_64-3.9/backports/lzma/_lzmamodule.o
x86_64-pc-linux-gnu-gcc-9.3.0 -pthread -shared -Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu -L. build/temp.linux-x86_64-3.9/backports/lzma/_lzmamodule.o -L/usr/lib/python-exec/python3.9/../../../lib -L/home/mgorny/lib -L/opt/local/lib -L/usr/local/lib -L/usr/lib64 -llzma -o build/lib.linux-x86_64-3.9/backports/lzma/_lzma.cpython-39-x86_64-linux-gnu.so
/usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: skipping incompatible /usr/lib/python-exec/python3.9/../../../lib/liblzma.so when searching for -llzma
/usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: skipping incompatible /usr/lib/python-exec/python3.9/../../../lib/libpthread.so when searching for -lpthread
/usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: skipping incompatible /usr/lib/python-exec/python3.9/../../../lib/libpthread.a when searching for -lpthread
/usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: skipping incompatible /usr/lib/python-exec/python3.9/../../../lib/libc.so when searching for -lc
/usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: skipping incompatible /usr/lib/python-exec/python3.9/../../../lib/libc.a when searching for -lc

but LLD fails entirely:

$ LDFLAGS='-fuse-ld=lld' python setup.py build
This is backports.lzma version 0.0.14
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.9
creating build/lib.linux-x86_64-3.9/backports
copying backports/__init__.py -> build/lib.linux-x86_64-3.9/backports
creating build/lib.linux-x86_64-3.9/backports/lzma
copying backports/lzma/__init__.py -> build/lib.linux-x86_64-3.9/backports/lzma
running build_ext
building 'backports.lzma._lzma' extension
creating build/temp.linux-x86_64-3.9
creating build/temp.linux-x86_64-3.9/backports
creating build/temp.linux-x86_64-3.9/backports/lzma
x86_64-pc-linux-gnu-gcc-9.3.0 -pthread -fPIC -I/usr/lib/python-exec/python3.9/../../../include -I/home/mgorny/include -I/opt/local/include -I/usr/local/include -I/usr/include/python3.9 -c backports/lzma/_lzmamodule.c -o build/temp.linux-x86_64-3.9/backports/lzma/_lzmamodule.o
x86_64-pc-linux-gnu-gcc-9.3.0 -pthread -shared -Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu -L. -fuse-ld=lld build/temp.linux-x86_64-3.9/backports/lzma/_lzmamodule.o -L/usr/lib/python-exec/python3.9/../../../lib -L/home/mgorny/lib -L/opt/local/lib -L/usr/local/lib -L/usr/lib64 -llzma -o build/lib.linux-x86_64-3.9/backports/lzma/_lzma.cpython-39-x86_64-linux-gnu.so
ld.lld: error: /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/../../../../lib64/crti.o is incompatible with elf_x86_64
ld.lld: error: /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/crtbeginS.o is incompatible with elf_x86_64
ld.lld: error: build/temp.linux-x86_64-3.9/backports/lzma/_lzmamodule.o is incompatible with elf_x86_64
ld.lld: error: /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/libgcc_s.so.1 is incompatible with elf_x86_64
ld.lld: error: /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/libgcc_s.so.1 is incompatible with elf_x86_64
ld.lld: error: /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/crtendS.o is incompatible with elf_x86_64
ld.lld: error: /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/../../../../lib64/crtn.o is incompatible with elf_x86_64
collect2: error: ld returned 1 exit status
error: command '/usr/bin/x86_64-pc-linux-gnu-gcc-9.3.0' failed with exit code 1

format argument not working

When using lzma.compress, it doesn't seem possible to pass the format argument. For example:
lzma.compress(data, format=FORMAT_RAW)
yields:
NameError: name 'FORMAT_RAW' is not defined

Same for the default, FORMAT_XZ:
NameError: name 'FORMAT_XZ' is not defined

Compilation error in python 2.7.2

One of our users got the error - https://dpaste.de/uztv Which says:

  gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/root/include -I/opt/local/include -I/usr/local/include -I/usr/include/python2.7 -c backports/lzma/_lzmamodule.c -o build/temp.linux-x86_64-2.7/backports/lzma/_lzmamodule.o
  gcc -pthread -shared -Wl,-z,relro build/temp.linux-x86_64-2.7/backports/lzma/_lzmamodule.o -L/root/lib -L/opt/local/lib -L/usr/local/lib -L/usr/lib64 -llzma -lpython2.7 -o build/lib.linux-x86_64-2.7/backports/lzma/_lzma.so
  /bin/ld: /usr/local/lib/libpython2.7.a(abstract.o): relocation R_X86_64_32S against `_Py_NotImplementedStruct' can not be used when making a shared object; recompile with -fPIC
  /usr/local/lib/libpython2.7.a: error adding symbols: Bad value
  collect2: error: ld returned 1 exit status
  error: command 'gcc' failed with exit status 1

This error was got when using python 2.7.2 . When they updated their python version, the compilation worked fine. It seems like the python 2.7.2 libpython2.7.a file does not have some symbols that later versions have.

peterjc / backports.lzma Goto Github PK

backports.lzma's People

Contributors

Stargazers

Watchers

Forkers

backports.lzma's Issues

Recommend Projects

Recommend Topics

Recommend Org