Comments (8)
Ok, some ctypes hackery to implement sys.orig_argv
for CPython<3.10
gets this1 working for all CPythons Pex supports and PyPy>=3.10
. It will not work for the PyPy>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,<3.10
that Pex supports because I can find no way to access the original PyPy process argv for those PyPy releases.
Footnotes
-
Where "this" is more than the OP. It's support for
python -u my.pex
as the basis for the OP request which will use a new--inject-python-args
option similar to the existing--inject-args
option (with no support for leakingPYTHON*
at all; i.e.: the ubiquitous-sE
used by Pex behind the scenes stays in force). ↩
from pex.
@bkad can you give a concrete example of what you're aiming for here? Passing flags to entry points is definitely supported already using --inject-args
. See this example use: https://docs.pex-tool.org/recipes.html#uvicorn-and-other-customizable-application-servers. What you can't do is pass flags to CPython itself when running an entry point (like, say -B
to skip bytecode generation). Is that the style of thing you want to do here?
from pex.
What you can't do is pass flags to CPython itself
This is what I'm trying to do, pass -u
to CPython. Alternatively, I could use the PYTHONUNBUFFERED environment variable, but I'd like to configure this within the PEX at build time rather than everywhere the PEX is executed. Passing PYTHONUNBUFFERED through --inject-env
has no effect on the re-executed interpreter for some reason.
from pex.
Ok. Yeah, you generally can't do this today. Only one incantation gets this to work, and then only after the 1st run:
- exe.py:
import sys if __name__ == "__main__": print(sys.stdout.buffer)
- Pex command:
:; pex --exe exe.py -o test.pex --python-shebang "/usr/bin/env python -u" --sh-boot
Nets:
:; ./test.pex
<_io.BufferedWriter name='<stdout>'>
:; ./test.pex
<_io.FileIO name='<stdout>' mode='wb' closefd=False>
:; ./test.pex
<_io.FileIO name='<stdout>' mode='wb' closefd=False>
...
Why this works is fairly arcane, but basically --sh-boot
respects args it finds in --python-shebang
(even if the --python-shebang specified is not valid!) and, after 1st boot when the PEX is laid out on disk under the PEX_ROOT, --sh-boot
optimizes by directly calling <full path to compatible python it discovered> <python args from --python-shebang> <installed PEX path/__main__.py> "$@"
. Notably, this only works for plain PEXes and not --venv
PEXes. Since most PEX users probably want both --sh-boot
and --venv
for minimum boot latency and max Python ecosystem compatibility, the example incantation is not generally viable.
In summary, @bkad this seems like a good feature to support across all PEX styles.
from pex.
For a bit more insight on the --sh-boot
machinery, using the example test.pex above:
:; head -123 test.pex
#!/bin/sh
# N.B.: This script should stick to syntax defined for POSIX `sh` and avoid non-builtins.
# See: https://pubs.opengroup.org/onlinepubs/9699919799/idx/shell.html
set -eu
VENV=""
VENV_PYTHON_ARGS="-sE"
# N.B.: This ensures tilde-expansion of the DEFAULT_PEX_ROOT value.
DEFAULT_PEX_ROOT="$(echo ~/.pex)"
DEFAULT_PYTHON="/usr/bin/env"
DEFAULT_PYTHON_ARGS="python -u"
PEX_ROOT="${PEX_ROOT:-${DEFAULT_PEX_ROOT}}"
INSTALLED_PEX="${PEX_ROOT}/unzipped_pexes/466b0553ddeea983208b65ba0b5f9870e10b1127"
if [ -n "${VENV}" -a -x "${INSTALLED_PEX}" ]; then
# We're a --venv execution mode PEX installed under the PEX_ROOT and the venv
# interpreter to use is embedded in the shebang of our venv pex script; so just
# execute that script directly.
export PEX="$0"
if [ -n "${VENV_PYTHON_ARGS}" ]; then
exec "${INSTALLED_PEX}/bin/python" "${VENV_PYTHON_ARGS}" "${INSTALLED_PEX}" \
"$@"
else
exec "${INSTALLED_PEX}/bin/python" "${INSTALLED_PEX}" "$@"
fi
fi
find_python() {
for python in \
"python3.12" \
"python3.11" \
"python3.10" \
"python3.9" \
"python3.8" \
"python3.7" \
"python3.6" \
"python3.5" \
"python2.7" \
"pypy3.12" \
"pypy3.11" \
"pypy3.10" \
"pypy3.9" \
"pypy3.8" \
"pypy3.7" \
"pypy3.6" \
"pypy3.5" \
"pypy2.7" \
"python3" \
"python2" \
"pypy3" \
"pypy2" \
"python" \
"pypy" \
; do
if command -v "${python}" 2>/dev/null; then
return
fi
done
}
if [ -x "${DEFAULT_PYTHON}" ]; then
python_exe="${DEFAULT_PYTHON} ${DEFAULT_PYTHON_ARGS}"
else
python_exe="$(find_python)"
fi
if [ -n "${python_exe}" ]; then
if [ -n "${PEX_VERBOSE:-}" ]; then
echo >&2 "$0 used /bin/sh boot to select python: ${python_exe} for re-exec..."
fi
if [ -z "${VENV}" -a -e "${INSTALLED_PEX}" ]; then
# We're a --zipapp execution mode PEX installed under the PEX_ROOT with a
# __main__.py in our top-level directory; so execute Python against that
# directory.
export __PEX_EXE__="$0"
exec ${python_exe} "${INSTALLED_PEX}" "$@"
else
# The slow path: this PEX zipapp is not installed yet. Run the PEX zipapp so it
# can install itself, rebuilding its fast path layout under the PEX_ROOT.
if [ -n "${PEX_VERBOSE:-}" ]; then
echo >&2 "Running zipapp pex to lay itself out under PEX_ROOT."
fi
exec ${python_exe} "$0" "$@"
fi
fi
echo >&2 "Failed to find any of these python binaries on the PATH:"
for python in \
"python3.12" \
"python3.11" \
"python3.10" \
"python3.9" \
"python3.8" \
"python3.7" \
"python3.6" \
"python3.5" \
"python2.7" \
"pypy3.12" \
"pypy3.11" \
"pypy3.10" \
"pypy3.9" \
"pypy3.8" \
"pypy3.7" \
"pypy3.6" \
"pypy3.5" \
"pypy2.7" \
"python3" \
"python2" \
"pypy3" \
"pypy2" \
"python" \
"pypy" \
; do
echo >&2 "${python}"
done
echo >&2 "Either adjust your $PATH which is currently:"
echo >&2 "${PATH}"
echo >&2 -n "Or else install an appropriate Python that provides one of the binaries in "
echo >&2 "this list."
exit 1
P!
.bootstrap/P!.bootstrap/pex/P!x�_ek␦.bootstrap/pex/__init__.pySVp�/�,�L�(Q0204H�P((��JM.QH��+)�L*-�/*��RV��LN�+NMQ(�KI-R(�HUp,HLRP��Ԣ���<#=��T�����_���P!(�e��␦.bootstrap/pex/__main__.py5��
Basically all the Pex python bootstrap logic is replaced with shell to drop boot overhead from ~50ms to ~1ms (Python is SLOW).
from pex.
Noting that the trick to all this will be protecting -sE
which is the standard set of Pex python args used to enforce the hermeticity guarantees of PEX files (the -E
is only meant to turn off PYTHONPATH
but it also turns off things like PYTHONUNBUFFERED
). ~All Python flags besides -s
and ~all PYTHON*
env vars besides PYTHONPATH
are likely safe to let propagate to the final exec
'd process.
from pex.
Alrighty, this is now released on 2.4.0,: https://github.com/pex-tool/pex/releases/tag/v2.4.0
Thanks @bkad.
from pex.
Much appreciated!
from pex.
Related Issues (20)
- ValueError: invalid literal for int() with base 10 during pex build HOT 5
- How to bootstrap external pex files within running Python interpreter HOT 23
- Finalize Python 3.13 support HOT 1
- Build fails/incomplete on requirements.txt schema not followed correctly HOT 9
- Allow subsetting a PEX lockfile in PEX format, not just pip HOT 12
- Support for pip-compile generated requirements with hashes HOT 8
- bdist_pex called from tox fails with python 3.12 HOT 7
- Regression of artifact URL due to change since pip 23.3 HOT 12
- Monkey patch warning when using PEX to run a gunicorn app HOT 6
- `pex3 lock export` can turn VCS requirements into PyPI ones, resulting in hash failures HOT 3
- Add support for `--override`ing transitive dependencies. HOT 1
- Add `--provided`: like `--exclude`, but require the excluded project is available at boot time.
- "Resolving wheels disallowed" error when using `--lock` and `--only-binary` HOT 3
- Add more distribution metadata parsing context to failures.
- Add support for Pip 24.2 / statically linked musl libc CPythons
- Adding support to have all command line flags respected via environment variables HOT 6
- How does pex resolve defaut `--pip-version` HOT 13
- Support `--scie-pbs-flavor`. HOT 3
- Support `--scie` for PyPy.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pex.