Giter VIP home page Giter VIP logo

Comments (8)

jsirois avatar jsirois commented on August 13, 2024 1

Ok, some ctypes hackery to implement sys.orig_argv for CPython<3.10 gets this1 working for all CPythons Pex supports and PyPy>=3.10. It will not work for the PyPy>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,<3.10 that Pex supports because I can find no way to access the original PyPy process argv for those PyPy releases.

Footnotes

  1. Where "this" is more than the OP. It's support for python -u my.pex as the basis for the OP request which will use a new --inject-python-args option similar to the existing --inject-args option (with no support for leaking PYTHON* at all; i.e.: the ubiquitous -sE used by Pex behind the scenes stays in force).

from pex.

jsirois avatar jsirois commented on August 13, 2024

@bkad can you give a concrete example of what you're aiming for here? Passing flags to entry points is definitely supported already using --inject-args. See this example use: https://docs.pex-tool.org/recipes.html#uvicorn-and-other-customizable-application-servers. What you can't do is pass flags to CPython itself when running an entry point (like, say -B to skip bytecode generation). Is that the style of thing you want to do here?

from pex.

bkad avatar bkad commented on August 13, 2024

What you can't do is pass flags to CPython itself

This is what I'm trying to do, pass -u to CPython. Alternatively, I could use the PYTHONUNBUFFERED environment variable, but I'd like to configure this within the PEX at build time rather than everywhere the PEX is executed. Passing PYTHONUNBUFFERED through --inject-env has no effect on the re-executed interpreter for some reason.

from pex.

jsirois avatar jsirois commented on August 13, 2024

Ok. Yeah, you generally can't do this today. Only one incantation gets this to work, and then only after the 1st run:

  • exe.py:
    import sys
    
    
    if __name__ == "__main__":
        print(sys.stdout.buffer)
  • Pex command:
    :; pex --exe exe.py -o test.pex --python-shebang "/usr/bin/env python -u" --sh-boot
    

Nets:

:; ./test.pex
<_io.BufferedWriter name='<stdout>'>
:; ./test.pex
<_io.FileIO name='<stdout>' mode='wb' closefd=False>
:; ./test.pex
<_io.FileIO name='<stdout>' mode='wb' closefd=False>
...

Why this works is fairly arcane, but basically --sh-boot respects args it finds in --python-shebang (even if the --python-shebang specified is not valid!) and, after 1st boot when the PEX is laid out on disk under the PEX_ROOT, --sh-boot optimizes by directly calling <full path to compatible python it discovered> <python args from --python-shebang> <installed PEX path/__main__.py> "$@". Notably, this only works for plain PEXes and not --venv PEXes. Since most PEX users probably want both --sh-boot and --venv for minimum boot latency and max Python ecosystem compatibility, the example incantation is not generally viable.

In summary, @bkad this seems like a good feature to support across all PEX styles.

from pex.

jsirois avatar jsirois commented on August 13, 2024

For a bit more insight on the --sh-boot machinery, using the example test.pex above:

:; head -123 test.pex
#!/bin/sh
# N.B.: This script should stick to syntax defined for POSIX `sh` and avoid non-builtins.
# See: https://pubs.opengroup.org/onlinepubs/9699919799/idx/shell.html
set -eu

VENV=""
VENV_PYTHON_ARGS="-sE"

# N.B.: This ensures tilde-expansion of the DEFAULT_PEX_ROOT value.
DEFAULT_PEX_ROOT="$(echo ~/.pex)"

DEFAULT_PYTHON="/usr/bin/env"
DEFAULT_PYTHON_ARGS="python -u"

PEX_ROOT="${PEX_ROOT:-${DEFAULT_PEX_ROOT}}"
INSTALLED_PEX="${PEX_ROOT}/unzipped_pexes/466b0553ddeea983208b65ba0b5f9870e10b1127"

if [ -n "${VENV}" -a -x "${INSTALLED_PEX}" ]; then
    # We're a --venv execution mode PEX installed under the PEX_ROOT and the venv
    # interpreter to use is embedded in the shebang of our venv pex script; so just
    # execute that script directly.
    export PEX="$0"
    if [ -n "${VENV_PYTHON_ARGS}" ]; then
        exec "${INSTALLED_PEX}/bin/python" "${VENV_PYTHON_ARGS}" "${INSTALLED_PEX}" \
            "$@"
    else
        exec "${INSTALLED_PEX}/bin/python" "${INSTALLED_PEX}" "$@"
    fi
fi

find_python() {
    for python in \
"python3.12" \
"python3.11" \
"python3.10" \
"python3.9" \
"python3.8" \
"python3.7" \
"python3.6" \
"python3.5" \
"python2.7" \
"pypy3.12" \
"pypy3.11" \
"pypy3.10" \
"pypy3.9" \
"pypy3.8" \
"pypy3.7" \
"pypy3.6" \
"pypy3.5" \
"pypy2.7" \
"python3" \
"python2" \
"pypy3" \
"pypy2" \
"python" \
"pypy" \
    ; do
        if command -v "${python}" 2>/dev/null; then
            return
        fi
    done
}

if [ -x "${DEFAULT_PYTHON}" ]; then
    python_exe="${DEFAULT_PYTHON} ${DEFAULT_PYTHON_ARGS}"
else
    python_exe="$(find_python)"
fi
if [ -n "${python_exe}" ]; then
    if [ -n "${PEX_VERBOSE:-}" ]; then
        echo >&2 "$0 used /bin/sh boot to select python: ${python_exe} for re-exec..."
    fi
    if [ -z "${VENV}" -a -e "${INSTALLED_PEX}" ]; then
        # We're a --zipapp execution mode PEX installed under the PEX_ROOT with a
        # __main__.py in our top-level directory; so execute Python against that
        # directory.
        export __PEX_EXE__="$0"
        exec ${python_exe} "${INSTALLED_PEX}" "$@"
    else
        # The slow path: this PEX zipapp is not installed yet. Run the PEX zipapp so it
        # can install itself, rebuilding its fast path layout under the PEX_ROOT.
        if [ -n "${PEX_VERBOSE:-}" ]; then
            echo >&2 "Running zipapp pex to lay itself out under PEX_ROOT."
        fi
        exec ${python_exe} "$0" "$@"
    fi
fi

echo >&2 "Failed to find any of these python binaries on the PATH:"
for python in \
"python3.12" \
"python3.11" \
"python3.10" \
"python3.9" \
"python3.8" \
"python3.7" \
"python3.6" \
"python3.5" \
"python2.7" \
"pypy3.12" \
"pypy3.11" \
"pypy3.10" \
"pypy3.9" \
"pypy3.8" \
"pypy3.7" \
"pypy3.6" \
"pypy3.5" \
"pypy2.7" \
"python3" \
"python2" \
"pypy3" \
"pypy2" \
"python" \
"pypy" \
; do
    echo >&2 "${python}"
done
echo >&2 "Either adjust your $PATH which is currently:"
echo >&2 "${PATH}"
echo >&2 -n "Or else install an appropriate Python that provides one of the binaries in "
echo >&2 "this list."
exit 1
P!
  .bootstrap/P!.bootstrap/pex/P!x�_ek␦.bootstrap/pex/__init__.pySVp�/�,�L�(Q0204H�P((��JM.QH��+)�L*-/*��RV��LN�+NMQ(�KI-R(�HUp,HLRP��Ԣ���<#=��T�����_���P!(�e��␦.bootstrap/pex/__main__.py5��

Basically all the Pex python bootstrap logic is replaced with shell to drop boot overhead from ~50ms to ~1ms (Python is SLOW).

from pex.

jsirois avatar jsirois commented on August 13, 2024

Noting that the trick to all this will be protecting -sE which is the standard set of Pex python args used to enforce the hermeticity guarantees of PEX files (the -E is only meant to turn off PYTHONPATH but it also turns off things like PYTHONUNBUFFERED). ~All Python flags besides -s and ~all PYTHON* env vars besides PYTHONPATH are likely safe to let propagate to the final exec'd process.

from pex.

jsirois avatar jsirois commented on August 13, 2024

Alrighty, this is now released on 2.4.0,: https://github.com/pex-tool/pex/releases/tag/v2.4.0

Thanks @bkad.

from pex.

bkad avatar bkad commented on August 13, 2024

Much appreciated!

from pex.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.