Giter VIP home page Giter VIP logo

pyeasyarchive's Introduction

Build Status Coverage Status

Introduction

A ctypes-based adapter to libarchive. The source-code is written to be clear and intuitive.

Even 7-Zip is supported for both reading and writing.

I could definitely use some help, if any is available. Completeness will require a bit more work (see libarchive's archive.h and archive_entry.h).

Installation

PyPI:

$ pip install libarchive

Notes

  • The Ubuntu libarchive package maintainer only provides a "libarchive.so" symlink in the dev package so you'll have to install the libarchive-dev package.
$ apt-get install libarchive-dev
  • Encryption is not currently supported since it's not supported in the underlying library (libarchive). Note this inquiry and the wishlist item.

  • OS X has a system version of libarchive that is very old. As a result, many users have encountered issues importing an alternate one. Specifically, often they install a different one via Brew but this will not be [sym]linked into the system like other packages. This is a precaution taken by Brew to prevent undefined behavior in the parts of OS X that depend on the factory version. In order to work around this, you should set LD_LIBRARY_PATH (or prepend if LD_LIBRARY_PATH is already defined) with the path of the location of the library version you want to use. You'll want to set this from your user-profile script (unless your environment can not support this and you need to prepend something like "LD_LIBRARY_PATH=/some/path" to the front of the command-line or set it via os.environ above where you import this package). A tool has been provided that will print the path of the first version of libarchive installed via Brew. Just copy-and-paste it. Thanks to @SkyLeach for discussing the issue and treatments.

Examples

To extract to the current directory from a physical file (and print each relative filepath):

import libarchive.public

for entry in libarchive.public.file_pour('/tmp/test.zip'):
    print(entry)

To extract to the current directory from memory:

import libarchive.public

with open('/tmp/test.zip', 'rb') as f:
    for entry in libarchive.public.memory_pour(f.read()):
        print(entry)

To read files from a physical archive:

import libarchive.public

with libarchive.public.file_reader('test.7z') as e:
    for entry in e:
        with open('/tmp/' + str(entry), 'wb') as f:
            for block in entry.get_blocks():
                f.write(block)

To read files from memory:

import libarchive.public

with open('test.7z', 'rb') as f:
    buffer_ = f.read()
    with libarchive.public.memory_reader(buffer_) as e:
        for entry in e:
            with open('/tmp/' + str(entry), 'wb') as f:
                for block in entry.get_blocks():
                    f.write(block)

To specify a format and/or filter for reads (rather than detecting it):

import libarchive.public
import libarchive.constants

with open('test.7z', 'rb') as f:
    buffer_ = f.read()
    with libarchive.public.memory_reader(
            buffer_,
            format_code=libarchive.constants.ARCHIVE_FORMAT_TAR_USTAR,
            filter_code=libarchive.constants.ARCHIVE_FILTER_GZIP
        ) as e:
        for entry in e:
            with open('/tmp/' + str(entry), 'wb') as f:
                for block in entry.get_blocks():
                    f.write(block)

To read the "filetype" flag for each entry:

import libarchive.public

with open('test.7z', 'rb') as f:
    buffer_ = f.read()
    with libarchive.public.memory_reader(f.read()) as e:
        for entry in e:
            print(entry.filetype)

The output of this is:

EntryFileType(IFREG=True, IFLNK=True, IFSOCK=True, IFCHR=False, IFBLK=False, IFDIR=False, IFIFO=False)
EntryFileType(IFREG=True, IFLNK=True, IFSOCK=True, IFCHR=False, IFBLK=False, IFDIR=False, IFIFO=False)
EntryFileType(IFREG=True, IFLNK=True, IFSOCK=True, IFCHR=False, IFBLK=False, IFDIR=False, IFIFO=False)

To create a physical archive from physical files:

import libarchive.public
import libarchive.constants

libarchive.public.create_file(
    'create.7z',
    libarchive.constants.ARCHIVE_FORMAT_7ZIP,
    ['/etc/profile']):

The path of the file to add will be recorded verbatim.

To create an archive in memory from physical files:

import libarchive.public
import libarchive.constants

with open('/tmp/new.7z', 'wb') as f:
    def writer(buffer_, length):
        f.write(buffer_)
        return length

    libarchive.public.create_generic(
        writer,
        format_name=libarchive.constants.ARCHIVE_FORMAT_7ZIP,
        files=['/etc/profile']):

Testing

libarchive uses nose/nose2 for testing:

tests$ ./run.py

pyeasyarchive's People

Contributors

ahmase avatar bovee avatar come-maiz avatar davemds avatar dsoprea avatar ml-bnr avatar wolph avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pyeasyarchive's Issues

cannot install libarchive on windows

Hello. I read #27 and followed the guide in the comments - #27 (comment)

Everything in the guide was accurate except for libnettle-7.dll was actually named libnettle-8.dll. I don't know that this difference would cause the issue or not.

So I try to install and get a message that libarchive.dll or it's dependencies are missing. libarchive.dll definately exists in the directory it's looking in.

(venv) C:\Users\ReenigneArcher\Documents\GitHub\RetroArcher.X>set Path=%PATH%;C:\Users\ReenigneArcher\Documents\GitH
ub\RetroArcher.X\lib\libarchive-3.5.1\bin

(venv) C:\Users\ReenigneArcher\Documents\GitHub\RetroArcher.X>pip install libarchive
Collecting libarchive
  Using cached libarchive-0.4.7.tar.gz (23 kB)
Requirement already satisfied: nose in c:\users\reenignearcher\documents\github\retroarcher.x\venv\lib\site-packages
 (from libarchive) (1.3.7)
Using legacy 'setup.py install' for libarchive, since package 'wheel' is not installed.
Installing collected packages: libarchive
    Running setup.py install for libarchive ... error
    ERROR: Command errored out with exit status 1:
     command: 'c:\users\reenignearcher\documents\github\retroarcher.x\venv\scripts\python.exe' -u -c 'import io, os,
 sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\ReenigneArcher\\AppData\\Local\\Temp\\pip-install-mq647264
\\libarchive_17b4382c4f6549b5a8ce70d3cc464a76\\setup.py'"'"'; __file__='"'"'C:\\Users\\ReenigneArcher\\AppData\\Loca
l\\Temp\\pip-install-mq647264\\libarchive_17b4382c4f6549b5a8ce70d3cc464a76\\setup.py'"'"';f = getattr(tokenize, '"'"
'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'
"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))'
 install --record 'C:\Users\ReenigneArcher\AppData\Local\Temp\pip-record-4f9t6itb\install-record.txt' --single-versi
on-externally-managed --compile --install-headers 'c:\users\reenignearcher\documents\github\retroarcher.x\venv\inclu
de\site\python3.9\libarchive'
         cwd: C:\Users\ReenigneArcher\AppData\Local\Temp\pip-install-mq647264\libarchive_17b4382c4f6549b5a8ce70d3cc4
64a76\
    Complete output (4 lines):
    running install
    error: Could not find module 'C:\Users\ReenigneArcher\Documents\GitHub\RetroArcher.X\lib\libarchive-3.5.1\bin\li
barchive.dll' (or one of its dependencies). Try using the full path with constructor syntax.
    Verifying that the library is accessible.
    Library can not be loaded: Could not find module 'C:\Users\ReenigneArcher\Documents\GitHub\RetroArcher.X\lib\lib
archive-3.5.1\bin\libarchive.dll' (or one of its dependencies). Try using the full path with constructor syntax.
    ----------------------------------------
ERROR: Command errored out with exit status 1: 'c:\users\reenignearcher\documents\github\retroarcher.x\venv\scripts\
python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\ReenigneArcher\\AppData\\
Local\\Temp\\pip-install-mq647264\\libarchive_17b4382c4f6549b5a8ce70d3cc464a76\\setup.py'"'"'; __file__='"'"'C:\\Use
rs\\ReenigneArcher\\AppData\\Local\\Temp\\pip-install-mq647264\\libarchive_17b4382c4f6549b5a8ce70d3cc464a76\\setup.p
y'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from
setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(
code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\ReenigneArcher\AppData\Local\Temp\pip-record-4f9t6itb\i
nstall-record.txt' --single-version-externally-managed --compile --install-headers 'c:\users\reenignearcher\document
s\github\retroarcher.x\venv\include\site\python3.9\libarchive' Check the logs for full command output.

TLDR; version

    Complete output (4 lines):
    running install
    error: Could not find module 'C:\Users\ReenigneArcher\Documents\GitHub\RetroArcher.X\lib\libarchive-3.5.1\bin\li
barchive.dll' (or one of its dependencies). Try using the full path with constructor syntax.
    Verifying that the library is accessible.
    Library can not be loaded: Could not find module 'C:\Users\ReenigneArcher\Documents\GitHub\RetroArcher.X\lib\lib
archive-3.5.1\bin\libarchive.dll' (or one of its dependencies). Try using the full path with constructor syntax.

image

How are parent directories in archives handled?

It's a common security issue with many archive packages that if the path within the archive is specified as ../../../something.txt you can accidently overwrite files outside of your working directory.

Does this library protect against that? If not, how can I manually specify where to write the output?

With the zipfile library I generally solve it like this:

import os


def sanitize_path(path):
    return os.path.normpath('/' + path).lstrip('/')


target_directory = 'something/'

with zipfile.ZipFile(zip_filename, 'r') as zip_fh:
    for name in zip_fh.namelist():
        clean_name = sanitize_path(name)
        extract_to = os.path.join(target_directory, clean_name)

        zip_.extract(
            name,
            extract_to,
        )

Bug in filetype detection for archive entries

The bug is in libarchive/adapters/archive_entry.py, in the ArchiveEntry class' filetype property, specifically line 99:

flags = dict([(k, (v & filetype) > 0)

This should change to:
flags = dict([(k, v == filetype)

The problem is that some of the IF* flags have overlapping bits set, so multiple flags will return (v & filetype) > 0 as being True. For example, IFREG = 0100000, IFLNK = 0120000, and IFSOCK = 0140000; so if the file is ANY one of these types, all 3 flags will be set.

ctypes.ArgumentError with file_reader on Python 3

With Python 3.4:

>>> with libarchive.public.file_reader("myfile.7z") as reader:
...     pass
... 
Traceback (most recent call last):
  File "…/venv/lib/python3.4/site-packages/libarchive/adapters/archive_read.py", line 63, in _archive_read_open_filename
    block_size_bytes)
ctypes.ArgumentError: argument 2: <class 'TypeError'>: wrong type

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.4/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "…/venv/lib/python3.4/site-packages/libarchive/adapters/archive_read.py", line 275, in _enumerator
    opener(archive_res)
  File "…/venv/lib/python3.4/site-packages/libarchive/adapters/archive_read.py", line 298, in opener
    _archive_read_open_filename(archive_res, filepath, block_size)
  File "…/venv/lib/python3.4/site-packages/libarchive/adapters/archive_read.py", line 66, in _archive_read_open_filename
    raise libarchive.exception.ArchiveError(message)
libarchive.exception.ArchiveError: None

It works fine with Python 2.7:

>>> with libarchive.public.file_reader("myfile.7z") as reader:
...     pass
... 
>>>

It seems to work if I force Python 3 to use bytes:

>>> with libarchive.public.file_reader(bytes("myfile.7z", "ascii")) as reader:
...     pass
...

Using unittest instead of nose doesn't work

As nose won't work with Python 3.9/3.10, I tried to use unittest instead, but some tests failed:

[    3s] test_enumerate_from_file (adapters.test_archive_read.TestArchiveRead) ... ok
[    3s] test_enumerate_from_memory (adapters.test_archive_read.TestArchiveRead) ... ok
[    3s] test_read_from_file (adapters.test_archive_read.TestArchiveRead) ... ok
[    3s] test_read_from_memory (adapters.test_archive_read.TestArchiveRead) ... ok
[    3s] test_read_symlinks (adapters.test_archive_read.TestArchiveRead) ... FAIL
[    3s] test_create_file (adapters.test_archive_write.TestArchiveWrite) ... ok
[    3s] test_create_file__unicode (adapters.test_archive_write.TestArchiveWrite) ... ok
[    3s] types.test_archive_entry (unittest.loader._FailedTest) ... ERROR
[    3s] 
[    3s] ======================================================================
[    3s] ERROR: types.test_archive_entry (unittest.loader._FailedTest)
[    3s] ----------------------------------------------------------------------
[    3s] ImportError: Failed to import test module: types.test_archive_entry
[    3s] Traceback (most recent call last):
[    3s]   File "/usr/lib64/python3.8/unittest/loader.py", line 436, in _find_test_path
[    3s]     module = self._get_module_from_name(name)
[    3s]   File "/usr/lib64/python3.8/unittest/loader.py", line 377, in _get_module_from_name
[    3s]     __import__(name)
[    3s] ModuleNotFoundError: No module named 'types.test_archive_entry'; 'types' is not a package
[    3s] 
[    3s] 
[    3s] ======================================================================
[    3s] FAIL: test_read_symlinks (adapters.test_archive_read.TestArchiveRead)
[    3s] ----------------------------------------------------------------------
[    3s] Traceback (most recent call last):
[    3s]   File "/home/abuild/rpmbuild/BUILD/libarchive-0.4.7/tests/adapters/test_archive_read.py", line 89, in test_read_symlinks
[    3s]     self.assertEqual(index, expected)
[    3s] AssertionError: {} != {'README.rst': 'libarchive/resources/README.rst'}
[    3s] - {}
[    3s] + {'README.rst': 'libarchive/resources/README.rst'}
[    3s] 
[    3s] ----------------------------------------------------------------------
[    3s] Ran 8 tests in 0.139s
[    3s] 
[    3s] FAILED (failures=1, errors=1)

Can't figure out how to handle .gz files with no content list (initrd.gz example)

test code:

    FILE="/Users/magregor/src/imgunbundle/test/test_data/isolinux/initrd.gz"
    import sys, os
    import pprint
    import libarchive.public as la
    with open(FILE, 'rb') as fh:
        with la.file_reader(fh.name) as lafh:
            count = 0
            for count,entry in enumerate(lafh):
                print str(entry)
                #print entry.filetype
                pprint.pprint(dir(entry.filetype))
            if not count:
                outfile = os.path.splitext(os.path.basename(FILE))[0]
                for entry in la.file_pour(FILE):
                    print(entry)
                print(outfile)
                raise Exception('No count')

result:

    Traceback (most recent call last):
      File "/Users/magregor/tmp/test_lafh.py", line 18, in <module>
        raise Exception('No count')
    Exception: No count

License file in pypi archive

The pypi source archive isn't including the LICENSE file. Would it be possible to add it? It is very helpful when packaging this for Linux distributions. Thank you.

Cannot create AR archive

I'm trying to create a custom deb package (ar format) and I'm receiving an error when setting the libarchive.constants.ARCHIVE_FORMAT_AR in libarchive.public.create_file.
The code is below:

import libarchive.constants
import libarchive.public

newDeb = ''.join(random.choice(string.ascii_lowercase + string.digits + string.ascii_uppercase) for _ in range(8))

tmpRan = ''.join(random.choice(string.ascii_lowercase + string.digits + string.ascii_uppercase) for _ in range(8))
tmpDir = '/tmp/' + tmpRan

newDeb = newDeb + ".deb"

for entry in libarchive.public.create_file('/tmp/'+newDeb,
                libarchive.constants.ARCHIVE_FORMAT_AR,
                [tmpDir+'/debian-binary', tmpDir+'/control.tar.gz', tmpDir+'/data.tar.gz' ]):
                print(entry)

the error is:

Traceback (most recent call last):
  File "test.py", line 429, in ar_files
    [tmpDir+'/debian-binary', tmpDir+'/control.tar.gz', tmpDir+'/data.tar.gz' ]):
  File "/usr/local/lib/python2.7/dist-packages/libarchive/adapters/archive_write.py", line 199, in _create
    _set_write_context(a, format_code, filter_code)
  File "/usr/local/lib/python2.7/dist-packages/libarchive/adapters/archive_write.py", line 184, in _set_write_context
    format_code)
  File "/usr/local/lib/python2.7/dist-packages/libarchive/adapters/archive_write_set_format.py", line 8, in archive_write_set_format
    code)
  File "/usr/local/lib/python2.7/dist-packages/libarchive/calls/archive_write_set_format.py", line 8, in _check_zero_success
    raise ValueError("Function returned failure: (%d)" % (value))
ValueError: Function returned failure: (-30)

If I change the format to libarchive.constants.ARCHIVE_FORMAT_7ZIP, everything goes well and it creates the archive.
Here are some info on my env:

root@ubuntu:/tmp# pip list | grep libarchive
libarchive (0.4.3)
root@ubuntu:/tmp# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.2 LTS
Release:    14.04
Codename:   trusty
root@ubuntu:/tmp# python -V
Python 2.7.6
root@ubuntu:/tmp# 

Setup.py (error)

pip 8.1.1 from c:\program files\python35\lib\site-packages (python 3.5)

Python 3.5.1 (v3.5.1:37a07cee5969, Dec 6 2015, 01:54:25) [MSC v.1900 64 bit (AMD64)] on win32

The (error) in question :

running install
Verifying that the library is accessible.
Library can not be loaded: [WinError 126] Le module spÚcifiÚ est introuvable
error: [WinError 126] Le module spÚcifiÚ est introuvable

I tried to comment those lines and install it without pip, and I get this (error?) :

byte-compiling C:\Program Files\Python35\Lib\site-packages\libarchive\constants\archive_entry.py >to archive_entry.cpython-35.pyc
File "C:\Program Files\Python35\Lib\site-packages\libarchive\constants\archive_entry.py", line 2
AE_IFMT = 0170000
............................^
SyntaxError: invalid token

What am I doing wrong?

sourcepath 50414b9

The raw in _archive_entry_sourcepath could be None.
The application of decode is then invalid.

In my use case, sourcepath is the only one returned None.

Cheers.

Install failed on windows

I'm getting the following error when trying to do pip install libarchive on win 10:
I have python 3.6 installed.

 Failed building wheel for libarchive
  Running setup.py clean for libarchive
Failed to build libarchive
Installing collected packages: libarchive
  Running setup.py install for libarchive ... error
    Complete output from command c:\users\s\appdata\local\programs\python\python36-32\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\S\\AppData\\Local\\Temp\\pip-build-qm7djpde\\libarchive\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record C:\Users\S\AppData\Local\Temp\pip-_pieybby-record\install-record.txt --single-version-externally-managed --compile:
    running install
    Verifying that the library is accessible.
    Library can not be loaded: [WinError 126] The specified module could not be found
    error: [WinError 126] The specified module could not be found

    ----------------------------------------
Command "c:\users\s\appdata\local\programs\python\python36-32\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\S\\AppData\\Local\\Temp\\pip-build-qm7djpde\\libarchive\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record C:\Users\S\AppData\Local\Temp\pip-_pieybby-record\install-record.txt --single-version-externally-managed --compile" failed with error code 1 in C:\Users\S\AppData\Local\Temp\pip-build-qm7djpde\libarchive\

import libarchive.public error AttributeError: dlsym(0x7fe20861a170, archive_read_open_filename_w): symbol not found

Similar to issue #16 but with a different function where it goes bad so raising this as a separate one just in case.

Been banging my head over this the entire day... and can't see it anymore. This doesn't seem to be a libarchive-c Python issue perse, but let's get it investigated and documented here because it seems a lot of users are stuck with this (according to my Google-fu [or lack thereof])

Environment:
MacOS 10.14.1
Python 3.7.0 using a venv with libarchive-c==2.8
libarchive installed via brew.
Used the brew_find_libarchive script to locate where the devil brew put it: /usr/local/Cellar/libarchive/3.3.3/lib

Then tried different iterations and variations of setting:

  • LD_LIBRARY_PATH both in my system environment variables and above the import using os.environ
  • LA_LIBRARY_FILEPATH both in my system environment variables and above the import using os.environ

Each and every time the error is:

>>> import libarchive.public
read format "cab" is not supported
read format "warc" is not supported
read format "7zip" is not supported
read format "rar" is not supported
read format "lha" is not supported
read filter "all" is not supported
read filter "rpm" is not supported
read filter "lzop" is not supported
read filter "lzma" is not supported
read filter "none" is not supported
read filter "compress" is not supported
read filter "lz4" is not supported
read filter "lzip" is not supported
read filter "bzip2" is not supported
read filter "xz" is not supported
read filter "uu" is not supported
read filter "gzip" is not supported
read filter "grzip" is not supported
read filter "lrzip" is not supported
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/cvanlabe/blablabla/venv/lib/python3.7/site-packages/libarchive/__init__.py", line 1, in <module>
    from .entry import ArchiveEntry
  File "/Users/cvanlabe/blablablavenv/lib/python3.7/site-packages/libarchive/entry.py", line 6, in <module>
    from . import ffi
  File "/Users/cvanlabe/blablabla/venv/lib/python3.7/site-packages/libarchive/ffi.py", line 184, in <module>
    c_int, check_int)
  File "/Users/cvanlabe/blablabla/venv/lib/python3.7/site-packages/libarchive/ffi.py", line 95, in ffi
    f = getattr(libarchive, 'archive_'+name)
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ctypes/__init__.py", line 369, in __getattr__
    func = self.__getitem__(name)
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ctypes/__init__.py", line 374, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(0x7fe20861a170, archive_read_open_filename_w): symbol not found

Any hints are more than welcome, mind that the steps outlined in issue #16 are already tried (and failed).

How can I help providing better info to troubleshoot this?

Support encrypted archives

As a stopgap measure, I would like for the documentation to be explicit about this limitation.

Adding "support reading encrypted archives" and "support writing encrypted archives" to the TODO list in README.rst would be a good first step towards improving visibility.

vnix$ python nst.py demo.zip
> {'_ArchiveEntryItReadable__is_consumed': False, '_ArchiveEntry__entry_res': c_void_p(34891728), '_ArchiveEntry__reader_res': 34889568, '_ArchiveEntry__is_consumed': False}
[moo.txt] SIZE=(4)
Traceback (most recent call last):
  File "nst.py", line 15, in <module>
    for entry in arch.file_pour(argv[1]):
  File "/home/tripleee/zip/env-libarchive/lib/python2.6/site-packages/libarchive/adapters/archive_read.py", line 384, in _pour
    "Pour failed: (%d) [%s]" % (r, message))
libarchive.exception.ArchiveError: Pour failed: (-25) [Encrypted file is unsupported]

__str__ returns bytes when casting entry to string

Hello,
I tried to use the libarchive wrapper with python 3 but seems like there is conversion problem since the str returns bytes and I wasn't able to decode with both 'utf-8' and 'ascii'. It seems the class also doesn't have any str behavior.

    def _decompress(self, filename):
        path = ('{}/{}'.format(self.dump_path, filename))
        with libarchive.public.file_reader(path) as compressed:
            for entry in compressed:
                with open('{}/{}'.format(self.dump_path, str(entry))) as f:
                    for block in entry.get_blocks():
                        f.write(block)
                # Assumption is made here that stackoverflow files contain a
                # signle file and that is the xml file we want to parse.
                return entry

custom read functions

Hello!

It would be great if this library supported custom read functions as supported in libarchive:

Sometimes, none of the packaged "open" functions will work for you. In that case, you can use the lower-level `archive_read_open` function. This accepts three callbacks and a pointer to your data:

- An open callback. This is legacy and is never necessary and should not be used.
- A read callback.
- A close callback.

Source

test_read_symlinks fails

Hello, I get a failure with this new test:

======================================================================
FAIL: test_read_symlinks (adapters.test_archive_read.TestArchiveRead)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/abuild/rpmbuild/BUILD/libarchive-0.4.6/tests/adapters/test_archive_read.py", line 89, in test_read_symlinks
    self.assertEquals(index, expected)
AssertionError: {} != {'README.rst': 'libarchive/resources/README.rst'}
- {}
+ {'README.rst': 'libarchive/resources/README.rst'}
-------------------- >> begin captured logging << --------------------
libarchive.adapters.archive_write: DEBUG: Opening archive (create).
libarchive.adapters.archive_write: DEBUG: Closing archive (create).
libarchive.adapters.archive_read: DEBUG: Enumerating through archive file: /tmp/tmp6yycgxpm/archive.7z
libarchive.adapters.archive_read: DEBUG: Archive read-format is: <ANY>
libarchive.adapters.archive_read: DEBUG: Archive read-filter is: <ANY>
libarchive.adapters.archive_read: DEBUG: Opening from file (file_enumerator): /tmp/tmp6yycgxpm/archive.7z
--------------------- >> end captured logging << ---------------------

I am running with

nosetests -s -v tests

with both python2 and python3 -- the result is the same for both. Have I missed something?

import libarchive.public error AttributeError: dlsym(0xXXXXXXX, archive_read_support_filter_all): symbol not found

Hi All,

I have installed libarchive using pip on Mac OS 10.11.6.

My Code:

import libarchive.public

When I run this I get this error

Traceback (most recent call last):
File "Test.py", line 1, in
import libarchive.public
File "/Library/Python/2.7/site-packages/libarchive/public.py", line 1, in
from libarchive.adapters.archive_read import
File "/Library/Python/2.7/site-packages/libarchive/adapters/archive_read.py", line 7, in
import libarchive.calls.archive_read
File "/Library/Python/2.7/site-packages/libarchive/calls/archive_read.py", line 17, in
c_archive_read_support_filter_all = libarchive.archive_read_support_filter_all
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ctypes/init.py", line 378, in getattr
func = self.getitem(name)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ctypes/init.py", line 383, in getitem
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(0x7fb1d1e47f60, archive_read_support_filter_all): symbol not found

I have installed libarchive also using brew.

Is this library supported on Mac ?

password encrypted zip files raise exception

afaict libarchive does support this, does pyeasyarchive provide a hook?

 '_asdict',
 '_fields',
 '_make',
 '_replace',
 'count',
 'index']
Traceback (most recent call last):
  File "/Users/magregor/tmp/test_lafh.py", line 22, in <module>
    for block in entry.get_blocks():
  File "/Users/magregor/.virtualenvs/imgunbundler/lib/python2.7/site-packages/libarchive/adapters/archive_read.py", line 219, in get_blocks
    for block in _read_by_block(self.reader_res):
  File "/Users/magregor/.virtualenvs/imgunbundler/lib/python2.7/site-packages/libarchive/adapters/archive_read.py", line 210, in _read_by_block
    (r,))
ValueError: Read failed (archive_read_data_block): (-30)```

RARV5 Support

Hi,

Reading RAR5 files doesn't seem to work. I keep getting a libarchive.exception.ArchiveError: b'Unrecognized archive format' error. I tried manually specifying the format, but it looks like there isn't a ARCHIVE_FORMAT_RAR_V5 constant in libarchive.constants.

Is RARV5 supported? I see it here, but not in the Python bindings.

Is it possible to update .7z archive in-place?

I'd like to try libarchive with .7z. But I need to update the archive inplace instead of copy the rest content from the old archive to the new archive then add the new file to the new archive. Is it possible to use libarchive to update .7z inplace?

c_ssize_t undefined

I am unable to import the library on Python 2.6.6.

(.env)[0] 09:31:14 tripleee@sow:/tmp$ git clone https://github.com/dsoprea/PyEasyArchive.git
Cloning into PyEasyArchive...
remote: Counting objects: 367, done.
remote: Total 367 (delta 0), reused 0 (delta 0), pack-reused 367
Receiving objects: 100% (367/367), 23.01 MiB | 5.14 MiB/s, done.
Resolving deltas: 100% (198/198), done.

(.env)[0] 09:31:28 tripleee@sow:/tmp$ cd PyEasyArchive/

(.env)[0] 09:31:44 tripleee@sow:/tmp/PyEasyArchive$ nosetests
EE
======================================================================
ERROR: Failure: NameError (name 'c_ssize_t' is not defined)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/tmp/.env/lib/python2.6/site-packages/nose/loader.py", line 414, in loadTestsFromName
    addr.filename, addr.module)
  File "/var/tmp/.env/lib/python2.6/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/var/tmp/.env/lib/python2.6/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/tmp/PyEasyArchive/tests/test_create.py", line 5, in <module>
    import libarchive.public
  File "/tmp/PyEasyArchive/libarchive/public.py", line 1, in <module>
    from libarchive.adapters.archive_read import \
  File "/tmp/PyEasyArchive/libarchive/adapters/archive_read.py", line 7, in <module>
    import libarchive.calls.archive_read
  File "/tmp/PyEasyArchive/libarchive/calls/archive_read.py", line 4, in <module>
    from libarchive.types.archive import *
  File "/tmp/PyEasyArchive/libarchive/types/archive.py", line 3, in <module>
    ARCHIVE_WRITE_CALLBACK = CFUNCTYPE(c_ssize_t, c_void_p, c_void_p, POINTER(c_void_p), c_size_t)
NameError: name 'c_ssize_t' is not defined
-------------------- >> begin captured logging << --------------------
libarchive.library: DEBUG: Using library: [libarchive.so]
--------------------- >> end captured logging << ---------------------

======================================================================
ERROR: Failure: NameError (name 'c_ssize_t' is not defined)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/tmp/.env/lib/python2.6/site-packages/nose/loader.py", line 414, in loadTestsFromName
    addr.filename, addr.module)
  File "/var/tmp/.env/lib/python2.6/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/var/tmp/.env/lib/python2.6/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/tmp/PyEasyArchive/tests/test_read.py", line 5, in <module>
    import libarchive.public
  File "/tmp/PyEasyArchive/libarchive/public.py", line 1, in <module>
    from libarchive.adapters.archive_read import \
  File "/tmp/PyEasyArchive/libarchive/adapters/archive_read.py", line 7, in <module>
    import libarchive.calls.archive_read
  File "/tmp/PyEasyArchive/libarchive/calls/archive_read.py", line 4, in <module>
    from libarchive.types.archive import *
  File "/tmp/PyEasyArchive/libarchive/types/archive.py", line 3, in <module>
    ARCHIVE_WRITE_CALLBACK = CFUNCTYPE(c_ssize_t, c_void_p, c_void_p, POINTER(c_void_p), c_size_t)
NameError: name 'c_ssize_t' is not defined

----------------------------------------------------------------------
Ran 2 tests in 0.046s

FAILED (errors=2)

Incidentally, e.g. http://nullege.com/codes/search/ctypes.c_ssize_t seems to suggest using the following idiom;

try:
    from ctypes import c_ssize_t
except ImportError:
    from ctypes import c_longlong as c_ssize_t

With that, I am able to get slightly further in loading the library, but not enough to actually use it.

Read failed (archive_read_data_block): (-30) while reading large archive by blocks

Download a large archive. Archive used in this example is a .7z archive from StackExchange dataset. It is 10.8Gb compressed and contains only one 55Gb file.

Read the file by blocks:

>>> import libarchive.public as libarchive
>>> with libarchive.file_reader('../dataset/stackoverflow.com-Posts.7z') as archive:
...     for entry in archive:
...         if str(entry) != 'Posts.xml':
...             continue
...         for block in entry.get_blocks():
...             pass
... 
Traceback (most recent call last):
  File "<stdin>", line 5, in <module>
  File "/usr/local/lib/python2.7/dist-packages/libarchive/adapters/archive_read.py", line 219, in get_blocks
    for block in _read_by_block(self.reader_res):
  File "/usr/local/lib/python2.7/dist-packages/libarchive/adapters/archive_read.py", line 210, in _read_by_block
    (r,))
ValueError: Read failed (archive_read_data_block): (-30)

setup.py : circular import

While installing libarchive I got the error :
Verifying that the library is accessible.Library can not be loaded: [WinError 126] Le module spécifié est introuvable.
I was surprised to discover that the line "import libarchive" is called within the file setup.py
I'm not aware of the reasons to change the name of the module but it's strange to import a module that you are trying to install or to install a module you are trying to import !

push to pypi

Looks like the version in pypi isn't the same as on here. Specifically, the entry constants in the pypi version only work for 2.7 and not 3+.

Invalid Token error

The installation process raises a SyntaxError in /libarchive/constants/archive_entry.py, line 1 as :

AE_IFMT = 0170000
^
SyntaxError: invalid token

handling symlinks

The filetype information returned is a bit confusing:

tmp/symlinktest/
{'IFBLK': True,
 'IFCHR': False,
 'IFDIR': True,
 'IFIFO': False,
 'IFLNK': False,
 'IFREG': False,
 'IFSOCK': True}
'Found dir'
tmp/symlinktest/hosts
{'IFBLK': True,
 'IFCHR': True,
 'IFDIR': False,
 'IFIFO': False,
 'IFLNK': True,
 'IFREG': True,
 'IFSOCK': True}
'Found link'
tmp/symlinktest/just_text.txt
{'IFBLK': False,
 'IFCHR': False,
 'IFDIR': False,
 'IFIFO': False,
 'IFLNK': True,
 'IFREG': True,
 'IFSOCK': True}
'Found regular file'

not only is IFLNK set for regular files as well as symlinks, but IFBLK seems to be the trigger if IFDIR is false for links.

In addition, I don't see a way to get the target of a symlink? simply opening and writing (as in the example) writes a normal 0-length file rather than a symlink.

installation fails on fresh 21.10 w/ "liblibarchive.so not found", both pip-repo as well as git+https from here

Hi,

installing on ubuntu 21.10 fails with some weird doubled "liblibarchive.so" error.
Happens both for the version from pip as well as directly from github:

    ERROR: Command errored out with exit status 1:
     command: /home/bf/eth/venv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-acvpa_1f/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-acvpa_1f/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-ud_9jj8l/install-record.txt --single-version-externally-managed --compile --install-headers /home/bf/eth/venv/include/site/python3.9/libarchive
         cwd: /tmp/pip-req-build-acvpa_1f/
    Complete output (4 lines):
    running install
    Verifying that the library is accessible.
    Library can not be loaded: [Errno 2] No such file or directory: b'liblibarchive.a'
    error: [Errno 2] No such file or directory: b'liblibarchive.a'
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/bf/eth/venv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-acvpa_1f/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-acvpa_1f/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-ud_9jj8l/install-record.txt --single-version-externally-managed --compile --install-headers /home/bf/eth/venv/include/site/python3.9/libarchive Check the logs for full command output.

(note the double "lib" in "liblibarchive".
libarchive.so
is installed:

 find / -xdev 2> /dev/null | grep -i libarchive.so
/usr/lib/x86_64-linux-gnu/libarchive.so.13
/usr/lib/x86_64-linux-gnu/libarchive.so.13.4.3
/usr/lib/x86_64-linux-gnu/libarchive.so

ARCHIVE_FORMAT_CPIO_SVR4_CRC not working

Hi,

I am basically trying to create a. Before i used cpio directly "cpio -ov -A -H crc -F" to create the archive, which works fine. Now I want to use libarchive from python to achieve the same result. Howver there are some issues with the Archive Format. With the format "ARCHIVE_FORMAT_CPIO" I can create the archive, but SWU-Update claims the archive has not the correct format. I think "ARCHIVE_FORMAT_CPIO_SVR4_CRC" needs to be used, but with this format, libarchive somehow exits with an error.

    for entry in libarchive.public.create_file(
            self._strUpdateFileName,
            libarchive.constants.ARCHIVE_FORMAT_CPIO_SVR4_CRC,
            lstCopyFile):
        pass
Traceback (most recent call last):
  File "swufilecreator.py", line 355, in <module>
    swuCreator.CreateUpdateFile('./output/')
  File "swufilecreator.py", line 100, in CreateUpdateFile
    strSha256 = swuCreatorRec.CreateUpdateFile(self._strBuildDirectory + '/')
  File "swufilecreator.py", line 140, in CreateUpdateFile
    self.__CreateOutputFile()
  File "swufilecreator.py", line 320, in __CreateOutputFile
    lstFilesToAdd):
  File "/usr/local/lib/python3.6/dist-packages/libarchive/adapters/archive_write.py", line 284, in create_file
    return _create(opener, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/libarchive/adapters/archive_write.py", line 198, in _create
    _set_write_context(a, format_code, filter_code)
  File "/usr/local/lib/python3.6/dist-packages/libarchive/adapters/archive_write.py", line 183, in _set_write_context
    format_code)
  File "/usr/local/lib/python3.6/dist-packages/libarchive/adapters/archive_write_set_format.py", line 6, in archive_write_set_format
    code)
  File "/usr/local/lib/python3.6/dist-packages/libarchive/calls/archive_write_set_format.py", line 8, in _check_zero_success
    raise ValueError("Function returned failure: (%d)" % (value))
ValueError: Function returned failure: (-30)

The same code works well to create an archive with the option "ARCHIVE_FORMAT_CPIO". I wonder what is the problem here?

Cannot import libarchive into Python 3.7.3 on Windows 10 version 1809. Support on Windows?

Not supported on Windows? At least state whether or not it is. This is what happens when I try to run "pip install libarchive" on Windows 10 with Python 3.7.3 from cmd prompt and git bash prompt:

(Two-Windows-Event-Log-Summarizers) C:\Coding\Python\PyVirtualEnvs\Two-Windows-Event-Log-Summarizers\Scripts>pip install libarchive
Collecting libarchive
Using cached https://files.pythonhosted.org/packages/b8/89/2952eefad82d9361f539de16ea85a2d98a7fb68eae2ba9ab0869aa2cee0f/libarchive-0.4.6.tar.gz
Requirement already satisfied: nose in c:\coding\python\pyvirtualenvs\two-windows-event-log-summarizers\lib\site-packages (from libarchive) (1.3.7)
Installing collected packages: libarchive
Running setup.py install for libarchive ... error
ERROR: Complete output from command 'C:\Coding\Python\PyVirtualEnvs\Two-Windows-Event-Log-Summarizers\Scripts\python.exe' -u -c 'import setuptools, tokenize;file='"'"'C:\Users\david\AppData\Local\Temp\pip-install-4zafasl3\libarchive\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record 'C:\Users\david\AppData\Local\Temp\pip-record-9_ciycqb\install-record.txt' --single-version-externally-managed --compile --install-headers 'C:\Coding\Python\PyVirtualEnvs\Two-Windows-Event-Log-Summarizers\include\site\python3.7\libarchive':
ERROR: running install
error: [WinError 126] The specified module could not be found
Verifying that the library is accessible.
Library can not be loaded: [WinError 126] The specified module could not be found
----------------------------------------

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.