Comments (13)
Pex is hermetic. It actively ignores the surrounding venv contents if run from a venv very much on purpose. This is perhaps more obvious if you don't run it from a venv but instead run the Pex PEX binary When you say pex --pip-version X
the X
must be from the list of versions Pex supports, and if its not the vendored version Pex ships with (A very old patched Pip 20.3.4), then Pex dog-foods itself to fetch the requested Pip (plus supporting setuptools and wheel) into a hermetic PEX under the PEX_ROOT (~/.pex/pip/<version>/pip.pex/...
by default).
If you haven't seen this, here's the list of supported versions as of Pex 2.13.0:
:; pex --help | grep -A5 "\--pip-version {"
--pip-version {latest,vendored,20.3.4-patched,22.2.2,22.3,22.3.1,23.0,23.0.1,23.1,23.1.1,23.1.2,23.2,23.3.1,23.3.2,24.0,24.1,24.1.1,24.1.2,24.2}
The version of Pip to use for resolving dependencies.
The `latest` version refers to the latest version in
this list (24.2) which is not necessarily the latest
Pip version released on PyPI. (default:
20.3.4-patched)
from pex.
So, to be more clear perhaps - unlike most Python tools, you can install Pex globally on your machine once. You can specify --python
, --interpreter-constraint
, etc to get it to look at any python on the system at any time using that one global install. Again, since Pex happens to be written in Python, this might not be obvious. But consider it written in another language. If it has to invoke Python code, it always does so in a hermetic subprocess. If you're familiar with uv
, like that - just slower ;).
from pex.
And as to why the --pip-version
defaults so old - Pex never breaks users - ever 1. You can still be on Python 2.7 and purposefully using the Pip 20.3.4 legacy resolver and you can still upgrade to Pex latest and have everything still work. This is great for backwards compatibility, but it means the Pex defaults are very often not what you want. It pays off to frequently re-read release notes and / or pex --help
as a result to see what new, better options may be available to improve resolution or PEX execution speed, etc.
Footnotes
from pex.
I wanted to confirm that it's not me doing something wrong, it's actually a common approach to "customize" pex with those options if I understood you correctly here
Yes. There are a ton of features hidden behind the 3 console scripts Pex ships with (pex
, pex3
and pex-tools
). Unfortunately, the only real documentation right now is the command line help. There are some very basic notes at https://docs.pex-tool.org and recipes as well, but the coverage is very poor.
are there maybe some public design docs there offer a glipse into the pex machinery?
Nope. Just the code and tinkering. A PEX is a zip, so unzip
it or use zipinfo
to list it, etc. You can use pex-tools ./my.pex ...
to further interrogate the PEX file itself. If you're curious you can learn a lot by just clearing the PEX_ROOT and then running a pex ...
command or running a PEX file and looking back in the PEX_ROOT to see the directory structure of what got populated (see : ~/.pex/{pip,installed_wheels,venvs,interpreters,...}
etc.)
from pex.
I actually was only installing it through pip, but it seems that if I build an isolate env, e.g. docker, I could download (can I?) and put a binary there instead of trying to put it into the global python site, which might be problematic (ubuntu doesn't like it)
Yes, you could install it in a myriad ways:
- Use the Pex PEX binary (requires the target system has an answer to
/usr/bin/env python
):curl -fL https://github.com/pex-tool/pex/releases/download/v2.13.0/pex > /some/path/pex chmod +x /some/path/pex export PATH=/some/path:$PATH
- Install in a venv:
python -mvenv pex.venv pex.venv/bin/pip install pex export PATH=pex.venv/bin:$PATH
- Install it using pipx if you happen to use pipx:
pipx install pex
- Build a hermetic Pex native binary - no Python on target system is required!
EXCEPT!, this doesn't work yet: #2484
# 1st get `pex` via one of the methods above! pex pex -c pex --venv --scie eager -o pex
from pex.
@maxkoretskyi your problem has nothing to do with Pex here, it has to do with your use of the subprocess
API. When you say subprocess.run(..., env={'PEX_VERBOSE': '3'})
you tell Python that {'PEX_VERBOSE': '3'}
is the complete and total environment to run the subprocess in. In other words, you nuke PATH
and HOME
and everything else from the environment the subprocess sees. Without PATH
, the /usr/bin/env
executable can't find python
. So you probably want: subprocess.run(..., env={**os.environ, 'PEX_VERBOSE': '3'})
from pex.
@maxkoretskyi I do, but I'd need to know more details. You could DM me at https://pex-tool.org/discord or email [email protected]. I also can provide recommendations if the work is not a good fit.
from pex.
@jsirois thanks a lot for the quick reply!
I wanted to confirm that it's not me doing something wrong, it's actually a common approach to "customize" pex with those options if I understood you correctly here:
You can specify --python, --interpreter-constraint, etc to get it to look at any python on the system at any time using that one global install.
unlike most Python tools, you can install Pex globally on your machine once.
I actually was only installing it through pip, but it seems that if I build an isolate env, e.g. docker, I could download (can I?) and put a binary there instead of trying to put it into the global python site, which might be problematic (ubuntu doesn't like it)
then Pex dog-foods itself to fetch the requested Pip
are there maybe some public design docs there offer a glipse into the pex machinery?
from pex.
@maxkoretskyi please let me know if you have any further questions. If not, I'd like to close this as an answered question.
from pex.
@jsirois yes, absolutely, thanks a for your elaborate answers!
from pex.
@jsirois need your help, if I install pex as binary like this
# installing pex directly as binary requires the `/usr/bin/env python` call to be available
RUN curl -fL https://github.com/pex-tool/pex/releases/download/v2.13.0/pex -o /usr/local/bin/pex
RUN chmod +x /usr/local/bin/pex
RUN chown ${uid}:${gid} /usr/local/bin/pex
the following simply code my.py
fails when run with python m.py
:
import subprocess
pex = ['/usr/local/bin/pex']
try:
subprocess.run(pex,check=True,capture_output=True,env={'PEX_VERBOSE': '3'})
except subprocess.CalledProcessError as exc:
error = f'Building pex failed with error: \n {exc.stderr.decode("unicode_escape")}'
print(error)
The error is:
/usr/bin/env: 'python': No such file or directory
which is weird, because /usr/bin/env python
is available:
$ /usr/bin/env python --version
Python 3.11.9
and if I run /usr/local/bin/pex
directly, not as a subprocess, all works OK
from pex.
@jsirois silly me! appreciate your insights, all is good now 🙋🏻
as a side note, do you by any chance do occasional consulting related to python? If not, maybe you know somebody good who does?
from pex.
Thanks! I'll send an email
from pex.
Related Issues (20)
- ValueError: invalid literal for int() with base 10 during pex build HOT 5
- How to bootstrap external pex files within running Python interpreter HOT 23
- Finalize Python 3.13 support HOT 1
- Build fails/incomplete on requirements.txt schema not followed correctly HOT 9
- Allow subsetting a PEX lockfile in PEX format, not just pip HOT 12
- Support for pip-compile generated requirements with hashes HOT 8
- bdist_pex called from tox fails with python 3.12 HOT 7
- Regression of artifact URL due to change since pip 23.3 HOT 12
- Monkey patch warning when using PEX to run a gunicorn app HOT 6
- `pex3 lock export` can turn VCS requirements into PyPI ones, resulting in hash failures HOT 3
- A way to configure sys.executable flags at PEX runtime HOT 8
- Add support for `--override`ing transitive dependencies. HOT 1
- Add `--provided`: like `--exclude`, but require the excluded project is available at boot time.
- "Resolving wheels disallowed" error when using `--lock` and `--only-binary` HOT 3
- Add more distribution metadata parsing context to failures.
- Add support for Pip 24.2 / statically linked musl libc CPythons
- Adding support to have all command line flags respected via environment variables HOT 6
- Support `--scie-pbs-flavor`. HOT 3
- Support `--scie` for PyPy.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pex.