Comments (32)
PR #21 up.
Please test. I did and it seems to work very well.
from pyeasyarchive.
@devssh, did you try something like this?
os.environ['LA_LIBRARY_FILEPATH'] = '/usr/local/Cellar/libarchive/3.3.2/lib/libarchive.dylib'
import libarchive.public
import libarchive.constants
from pyeasyarchive.
I solved this by compiling libarchive from source https://github.com/libarchive/libarchive rather than via brew.
from pyeasyarchive.
@rocky I hear that is dangerous. That's why homebrew doesn't fix the issue: it breaks other things on the mac.
This needs to be elevated to bug, because the osx libarchive is out of date and this module ignores the LDFLAGS and CPPFLAGS required to point it to a newer version of the library (/usr/local/opt/libarchive for instance, the location used by homebrew).
from pyeasyarchive.
This library just imports whatever is available. How are compiler flags supposed to remedy that? We're not compiling anything.
from pyeasyarchive.
@dsoprea I flagged this for looking into it this evening. I figured it was a dlopen call, not a compiler call.
from pyeasyarchive.
@SkyLeach In the case of OSX having such an old version, I'm not sure there's a better way to point to a different version than to just set LA_LIBRARY_FILEPATH
in the environment.
Thanks for bringing that reality to my attention.
from pyeasyarchive.
@dsoprea I wasn't aware of that environment variable (grepped for the more common LD_LIBRARY_PATH).
Any reason why modding library.py to examine both might be a bad idea? In my experience, everyone just assumes the common environment variables.
from pyeasyarchive.
from pyeasyarchive.
Ok, so I started looking at this. I'll issue a pull request when done if necessary, but I suspect the fault is actually in ctypes.
from pyeasyarchive.
Still looking at this, but the simple fix didn't work.
from pyeasyarchive.
Having the issue as well, let me know if you manage to find a fix @SkyLeach !
from pyeasyarchive.
I have a fix, working the PR now
from pyeasyarchive.
I didn't see any specific reasons why we can't just check for LD_LIBRARY_PATH
(the conventional variable) in addition to the current variable. This was my original suggestion. Wouldn't the correct solution be to add the correct location to this in the environment so that anything that seeks to use it will find the correct one (a global solution rather than one localized to this project)?
from pyeasyarchive.
If I'm not mistaken, this ONLY works during setup.
testing:
running install_egg_info
Copying libarchive.egg-info to /Users/magregor/.virtualenvs/PyEasyArchive-skyleach/lib/python2.7/site-packages/libarchive-0.4.3-py2.7.egg-info
running install_scripts
(PyEasyArchive-skyleach) ________________________________________________________________________________
| ~/src/PyEasyArchive @ MAGREGOR-M-W0XX (magregor)
| => python
Python 2.7.13 (default, Dec 18 2016, 07:03:39)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import libarchive
>>>
(PyEasyArchive-skyleach) ________________________________________________________________________________
| ~/src/PyEasyArchive @ MAGREGOR-M-W0XX (magregor)
| => python2.7 ~/tmp/test_lafh.py
Traceback (most recent call last):
File "/Users/magregor/tmp/test_lafh.py", line 8, in <module>
import libarchive.public as la
File "/Users/magregor/.virtualenvs/PyEasyArchive-skyleach/lib/python2.7/site-packages/libarchive/public.py", line 1, in <module>
from libarchive.adapters.archive_read import \
File "/Users/magregor/.virtualenvs/PyEasyArchive-skyleach/lib/python2.7/site-packages/libarchive/adapters/archive_read.py", line 7, in <module>
import libarchive.calls.archive_read
File "/Users/magregor/.virtualenvs/PyEasyArchive-skyleach/lib/python2.7/site-packages/libarchive/calls/archive_read.py", line 17, in <module>
c_archive_read_support_filter_all = libarchive.archive_read_support_filter_all
File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ctypes/__init__.py", line 375, in __getattr__
func = self.__getitem__(name)
File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ctypes/__init__.py", line 380, in __getitem__
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(0x7fda33d15d20, archive_read_support_filter_all): symbol not found
So I'm going to move the function out of the setup.py
from pyeasyarchive.
@SkyLeach Yes. If we're all convinced that this is the way to go, then it'd have to go in library.py.
from pyeasyarchive.
@dsoprea done
This will slow down library loading a tiny bit for homebrew users (only once when first imported though)
from pyeasyarchive.
@SkyLeach See my other inquiry from thirty-five minutes ago.
from pyeasyarchive.
@dsoprea sure we can, but it'll still only work for PyEasyArchive.
LD_LIBRARY_PATH wasn't finding libarchive.13.dylib (which is what goes to /usr/local/opt/libarchive
)
The only difference would be that the same code has to run, and then copen would have to search the new path for libarchive.dylib. Two searches instead of just the one, and no difference in overhead.
If we tried to check LD_LIBRARY_PATH then we'd have to worry about it being set for everything else, but not libarchive since it doesn't get installed in the normal place.
from pyeasyarchive.
I need you to walk me through this. As I understand it, there is already a system-global libarchive
and, if anyone wants a newer one, they'd install it via Brew. Obviously this one isn't Brew-linked because then it'd conflict with the system-global one. However, why can't you just prepend/add the Brew one to LD_LIBRARY_PATH
/LA_LIBRARY_FILEPATH
in the system environment or right above where you import this project?
I'm not concerned about discussing how libarchive
affects the rest of the system. It's sufficient to know just that there's already a system-global one that has to remain in-place for OS X.
Since this is an exclusive Brew/Mac problem, it's not appropriate to put conditional logic in the project that doesn't help anyone else, especially when that logic depends on third-party tooling. Rather, the proper method should be identified and then be added as a footnote in the documentation.
from pyeasyarchive.
@dsoprea this is an OS-specific issue. On OSX the libarchive shipped with the OS is required and old (doesn't work with PyEasyArchive and/or newer archives). You can't just replace it. Most mac users use homebrew. If not, then they will most likely have LD_LIBRARY_PATH set since they are doing something very custom. Simply setting LD_LIBRARY_PATH with a mac (default) or a mac (with homebrew) won't work, because of the above issues.
Certainly it is possible to add to the documentation that all mac users need to do some esoteric stuff (basically what I just did) in order to support libarchive and PyEasyArchive. It can be done in .bash_profile (or .bashrc) using similar commands, although much more complicated with bash to strip the version number. Not using the specific installed version (that I pull with brew info --json) means it will not be found by the find_library function.
Is this technically specific to mac + homebrew users? Yes. That crowd just happens to be 90% of mac users who do development, thus making it the lions share of anyone wishing to install PyEasyArchive on a mac.
from pyeasyarchive.
@SkyLeach (and everyone else)
I'm suggesting that the affected individuals set the value into LD_LIBRARY_PATH
using one of the three ways:
- Prepending the environment change to the command-line that they're executing the command-line from.
- Doing a call to
os.environ
above the import. - Setting it into their user-profile (which should only affect their terminal sessions, but I'm totally sure).
I added a clause to the "Notes" section of the document as well as a script, based on your code, to print the path.
Check it and then I'll merge.
Everyone else: Does this help? Is this or isn't this sufficient?
from pyeasyarchive.
Looks good, but I'll need to test to be sure that the system library isn't found before/instead-of the environment library (which was one issue).
from pyeasyarchive.
from pyeasyarchive.
@dsoprea it seems so, but doesn't work as intended. We may still need to tell users to use LA_LIBRARY_FILEPATH:
(PyEasyArchive-skyleach) ________________________________________________________________________________
| ~/src/PyEasyArchive @ MAGREGOR-M-W0XX (magregor)
| => LD_LIBRARY_PATH=/usr/local/Cellar/libarchive/3.2.2/lib python2.7 ~/tmp/test_lafh.py
Traceback (most recent call last):
File "/Users/magregor/tmp/test_lafh.py", line 8, in <module>
import libarchive.public as la
File "/Users/magregor/.virtualenvs/PyEasyArchive-skyleach/lib/python2.7/site-packages/libarchive/public.py", line 1, in <module>
from libarchive.adapters.archive_read import \
File "/Users/magregor/.virtualenvs/PyEasyArchive-skyleach/lib/python2.7/site-packages/libarchive/adapters/archive_read.py", line 7, in <module>
import libarchive.calls.archive_read
File "/Users/magregor/.virtualenvs/PyEasyArchive-skyleach/lib/python2.7/site-packages/libarchive/calls/archive_read.py", line 17, in <module>
c_archive_read_support_filter_all = libarchive.archive_read_support_filter_all
File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ctypes/__init__.py", line 375, in __getattr__
func = self.__getitem__(name)
File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ctypes/__init__.py", line 380, in __getitem__
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(0x7fc5e4c6f640, archive_read_support_filter_all): symbol not found
LA_LIBRARY_FILEPATH=/usr/local/Cellar/libarchive/3.2.2/lib/libarchive.dylib python2.7 ~/tmp/test_lafh.py
works fine though
from pyeasyarchive.
My test script btw (which is pretty simple):
FILE="dcnm-installer.6.1.2.iso"
import sys, os
import pprint
import libarchive.public as la
with open(FILE, 'rb') as fh:
with la.file_reader(fh.name) as lafh:
entries = False
for entry in lafh:
print str(entry)
pprint.pprint(dir(entry.filetype))
please disregard the open fh then using fh.name, I was using it to test for os filelocks
from pyeasyarchive.
It goes back to my original suggestion. ctypes
doesn't use LD_LIBRARY_PATH
. I've updated the library-load logic to implement it. Do an update and retry. Note that you can run "./run.py" from the "tests/" subdirectory to run a couple of tests to test the loading, too.
from pyeasyarchive.
I just went ahead and merged. It's already been tested.
Everyone: Please see if this makes it easier for you to cope with running under OS X.
from pyeasyarchive.
I tried running it and didn't work on mac High Sierra, libarchive v0.4.4
I just ran pip install libarchive
and import libarchive.public
gives error, same on python 2 as well
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/anaconda3/envs/py36/lib/python3.6/site-packages/libarchive/public.py", line 1, in <module>
from libarchive.adapters.archive_read import \
File "/usr/local/anaconda3/envs/py36/lib/python3.6/site-packages/libarchive/adapters/archive_read.py", line 7, in <module>
import libarchive.calls.archive_read
File "/usr/local/anaconda3/envs/py36/lib/python3.6/site-packages/libarchive/calls/archive_read.py", line 17, in <module>
c_archive_read_support_filter_all = libarchive.archive_read_support_filter_all
File "/usr/local/anaconda3/envs/py36/lib/python3.6/ctypes/__init__.py", line 361, in __getattr__
func = self.__getitem__(name)
File "/usr/local/anaconda3/envs/py36/lib/python3.6/ctypes/__init__.py", line 366, in __getitem__
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(0x7fee5f551c10, archive_read_support_filter_all): symbol not found
from pyeasyarchive.
It did not work, although I am unable to find the lib/libarchive.dylib as I am using Anaconda which might be another complication on top of MacOSX High Sierra
Requirement already satisfied: libarchive in /usr/local/anaconda3/envs/py36/lib/python3.6/site-packages
Requirement already satisfied: nose in /usr/local/anaconda3/envs/py36/lib/python3.6/site-packages (from libarchive)
I'll get back to you if I find it
from pyeasyarchive.
Today, I was trying to use this lib as opposed to libarchive-c, and stumbled across this very issue again.
Platform MacOS 10.14.4
brew 2.1.2
libarchive 3.3.3 (installed through brew)
The only way I got it work was setting this (and only this) variable: LA_LIBRARY_FILEPATH .
To launch Python with it:
$ LA_LIBRARY_FILEPATH=/usr/local/Cellar/libarchive/3.3.3/lib/libarchive.13.dylib python
Python 3.7.3 (default, Mar 27 2019, 09:23:15)
[Clang 10.0.1 (clang-1001.0.46.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import libarchive.public
>>>
Note that for libarchive-c (the other one project :) ) you need to set a different environment variable:
$ LIBARCHIVE=/usr/local/Cellar/libarchive/3.3.3/lib/libarchive.13.dylib python
Python 3.7.3 (default, Mar 27 2019, 09:23:15)
[Clang 10.0.1 (clang-1001.0.46.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import libarchive
>>>
I suggest the instructions in the Readme.md get updated with the above for OS X users, and this bug gets closed?
from pyeasyarchive.
I’m on macOS and I can confirm that this LA_LIBRARY_FILEPATH
solution is the only one that worked for me.
I also used Homebrew’s brew --prefix libarchive
command, that gives a future-proof prefix path for the library. The inconvenient of hardcoding the /usr/local/Cellar/...
path is it’ll break when libarchive is updated.
LA_LIBRARY_FILEPATH=$(brew --prefix libarchive)/lib/libarchive.dylib
from pyeasyarchive.
Related Issues (20)
- __str__ returns bytes when casting entry to string HOT 3
- import libarchive.public error AttributeError: dlsym(0x7fe20861a170, archive_read_open_filename_w): symbol not found HOT 3
- test_read_symlinks fails HOT 4
- UnicodeDecodeError: 'ascii' codec can't decode byte 0xb0 in position 2: ordinal not in range(128) HOT 7
- Cannot import libarchive into Python 3.7.3 on Windows 10 version 1809. Support on Windows? HOT 1
- Finish implementing write-options and read-options into adapters
- custom read functions
- Is it possible to update .7z archive in-place? HOT 1
- sourcepath 50414b9
- Not possible to get information about archive? (solid, ...) HOT 2
- Using unittest instead of nose doesn't work HOT 2
- RARV5 Support HOT 6
- cannot install libarchive on windows HOT 1
- compression compress big directory to multivolume
- Publish a pypi version (or set tag) whose license is changed to MIT
- installation fails on fresh 21.10 w/ "liblibarchive.so not found", both pip-repo as well as git+https from here HOT 5
- UnicodeDecodeError on files in archive if filename contains non unicode characters (ex: é à etc.)
- UnicodeDecodeError on files in archive if filename contains non-ASCII characters (e.g. CJK characters)
- setup.py : circular import
- ARCHIVE_FORMAT_CPIO_SVR4_CRC not working
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pyeasyarchive.