jaraco / pip-run Goto Github PK

View Code? Open in Web Editor NEW

118.0 118.0 21.0 703 KB

pip-run - dynamic dependency loader for Python

License: MIT License

Python 100.00%

pip-run's People

Contributors

Stargazers

Watchers

Forkers

anthrotype daukantas hagarwa3 benj95 benoit-pierre smheidrich pombredanne ravikant-trafers hixio-mh feryataydogan laurencewarne techcable bayanijulian sirosen angerson wormon kurtmckee bswck pfmoore

pip-run's Issues

TypeError: split_at() got an unexpected keyword argument 'maxsplit'

Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/localdata/some_user/src/test_utils/venv/lib/python3.8/site-packages/pip-run.py", line 4, in <module>
    __name__ == '__main__' and run()
  File "/localdata/some_user/src/test_utils/venv/lib/python3.8/site-packages/pip_run/__init__.py", line 15, in run
    pip_args, params = commands.parse_script_args(args)
  File "/localdata/some_user/src/test_utils/venv/lib/python3.8/site-packages/pip_run/commands.py", line 58, in parse_script_args
    return _separate_dash(args)
  File "/localdata/some_user/src/test_utils/venv/lib/python3.8/site-packages/pip_run/commands.py", line 48, in _separate_dash
    pre, post = split_at(args, '--'.__eq__, maxsplit=1)
TypeError: split_at() got an unexpected keyword argument 'maxsplit'

The maxsplit keyword argument for split_at was only introduced from version 8.3.0 of more-itertools

Since there is no minimum version specified in

pip-run/setup.cfg

Line 26 in c72c1dd

more_itertools

Then if the users environment already contains this module at a lower version, then they will encounter the error above

Suggest the requirement be updated to more_itertools>=8.3.0?

To replicate (Python3.8):

virtualenv venv -p python3 && source ./venv/bin/activate
pip install more_itertools==8.2.0
pip install pip-run
python3 -m pip-run some_url -- -m some_module
...
TypeError: split_at() got an unexpected keyword argument 'maxsplit'

Stdin not piped

When piping input to a rwt-launched process, the stdin is not piped so any content piped is lost at the rwt invocation.

DeprecationWarning in nbformat (tests)

I'm trying to package your module as an rpm package. So I'm using the typical build, install and test cycle used on building packages from non-root account.

"setup.py build"
"setup.py install --root </install/prefix>"
"pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>

Here is pytest output:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-pip-run-8.6.0-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-pip-run-8.6.0-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.12, pytest-6.2.5, py-1.10.0, pluggy-0.13.1
Using --randomly-seed=1944680334
rootdir: /home/tkloczko/rpmbuild/BUILD/pip-run-8.6.0, configfile: pytest.ini
plugins: forked-1.3.0, shutil-1.7.0, virtualenv-1.7.0, flake8-1.0.7, xdist-2.3.0, cov-2.12.1, mock-3.6.1, localserver-0.5.0, timeout-2.0.1, flaky-3.7.0, hypothesis-6.14.6, rerunfailures-9.1.1, checkdocs-2.7.1, anyio-3.3.1, tornasync-0.6.0.post2, randomly-3.8.0, freezegun-0.4.2, pylama-7.7.1, black-0.3.12, trio-0.7.0, twisted-1.13.3, asyncio-0.15.1
collected 32 items

pip_run/tests/test_launch.py ....x                                                                                                                                   [ 16%]
pip_run/tests/test_deps.py ..                                                                                                                                        [ 23%]
pip_run/deps.py .                                                                                                                                                    [ 26%]
pip_run/commands.py .                                                                                                                                                [ 30%]
pip_run/scripts.py .                                                                                                                                                 [ 33%]
. .                                                                                                                                                                  [ 36%]
pip_run/tests/test_deps.py .                                                                                                                                         [ 40%]
pip_run/read-deps.py .                                                                                                                                               [ 43%]
pip_run/deps.py .                                                                                                                                                    [ 46%]
pip_run/tests/test_deps.py .                                                                                                                                         [ 50%]
pip_run/text.py .                                                                                                                                                    [ 53%]
pip_run/tests/test_deps.py ..                                                                                                                                        [ 60%]
pip_run/tests/test_scripts.py ............                                                                                                                           [100%]

============================================================================= warnings summary =============================================================================
pip_run/tests/test_scripts.py::TestNotebookDepsReader::test_code_and_markdown
pip_run/tests/test_scripts.py::TestNotebookDepsReader::test_code_and_markdown
pip_run/tests/test_scripts.py::TestNotebookDepsReader::test_jupyter_directives
pip_run/tests/test_scripts.py::TestNotebookDepsReader::test_jupyter_directives
pip_run/tests/test_scripts.py::TestNotebookDepsReader::test_multiple_code_blocks
pip_run/tests/test_scripts.py::TestNotebookDepsReader::test_multiple_code_blocks
pip_run/tests/test_scripts.py::TestNotebookDepsReader::test_one_code_block
  /usr/lib/python3.8/site-packages/jsonschema/validators.py:190: DeprecationWarning: Passing a schema to Validator.iter_errors is deprecated and will be removed in a future release. Call validator.evolve(schema=new_schema).iter_errors(...) instead.
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/warnings.html
========================================================================= short test summary info ==========================================================================
XFAIL pip_run/tests/test_launch.py::test_with_path_overlay
  cleanup can't occur with execv; #4
================================================================ 29 passed, 1 xfailed, 7 warnings in 15.99s ================================================================

Install fails on Xenial

On Ubuntu Xenial with sudo aptitude install -y python3-pip, and python3 -m pip install --user rwt:

jaraco@dev-jaraco:~$ python3 -m rwt pymongo
Loading requirements using pymongo
Exception:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 209, in main
    status = self.run(options, args)
  File "/usr/lib/python3/dist-packages/pip/commands/install.py", line 335, in run
    prefix=options.prefix_path,
  File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 732, in install
    **kwargs
  File "/usr/lib/python3/dist-packages/pip/req/req_install.py", line 837, in install
    self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
  File "/usr/lib/python3/dist-packages/pip/req/req_install.py", line 1039, in move_wheel_files
    isolated=self.isolated,
  File "/usr/lib/python3/dist-packages/pip/wheel.py", line 247, in move_wheel_files
    prefix=prefix,
  File "/usr/lib/python3/dist-packages/pip/locations.py", line 153, in distutils_scheme
    i.finalize_options()
  File "/usr/lib/python3.5/distutils/command/install.py", line 273, in finalize_options
    raise DistutilsOptionError("can't combine user with prefix, "
distutils.errors.DistutilsOptionError: can't combine user with prefix, exec_prefix/home, or install_(plat)base
Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/jaraco/.local/lib/python3.5/site-packages/rwt/__main__.py", line 12, in <module>
    with deps.load(*pip_args) as home:
  File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/home/jaraco/.local/lib/python3.5/site-packages/rwt/deps.py", line 42, in load
    subprocess.check_call(cmd)
  File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('/usr/bin/python3', '-m', 'pip', 'install', '-q', '-t', '/tmp/rwt-sp95m3xg', 'pymongo')' returned non-zero exit status 2

Emitting pip installer output by default is irritating

In pypa/pip#3971 (comment), @pfmoore characterizes the issue:

I'm not in love with details like the default being to print all of pip's output, [contributing] to a general sense of "I never think to use it, because it's sort of alright, but irritating". What I want is to be able to type pip-run my_script.py arg1 arg2 arg3, and have it run my script with all dependencies available. The other features of pip-run are secondary (to me).

It is often super-useful to have the output emitted, because it provides a report of what was installed. This is great when copy/pasting for a bug report, because it provides a snapshot of the relevant package versions (and method of installation).

I've also bemoaned the fact that the output includes pip installer output by default. I'd very much prefer to suppress that output by default, but make it an option to re-enable.

Because that's not the case, I often invoke pip-run with -q to silence that behavior (which adds another annoyance to the experience, similar to #57. There's no clean, quiet experience.

The reason the behavior is because that's the default pip install behavior and pip-run is trying not to re-invent the interface but instead loosely couple with the existing interface. In that way, many other installer options are readily available, such as -v if you need more detail about why the install is failing, or --index or --use-pep517. Without that generalization, it would be a maintenance challenge (nightmare?) to keep up with the mapping between the pip-run UX and the pip install UX.

I'd be interested in making the install 'quiet by default', but I'm not sure how to do that safely and still honor the flags to pip install.

For instance, if pip-run were to set PIP_QUIET=1, does that have the intended effect? Can the user override that with -v?

I ran some quick tests and at first blush, it looks like everything works as one might expect, with the installer details being suppressed by default, but re-activated with -v, and -vvv has the same effect as -vv without PIP_QUIET.

So I think this one has an easy fix - set PIP_QUIET when invoking pip install.

pip_run.read-deps should error when given a nonexistent/unreadable file

Currently, when pip_run.read-deps is passed a filename that does not exist or cannot be opened, it simply prints an empty line and exits successfully. I contend that this is the wrong behavior; such a filename is an error and should be treated as an error; an error message should be printed, and the command should exit nonzero.

Add PEP 518 build-requires support

Although the PEP 518 support added by pip 10 goes a long way toward supporting the use cases of installing dependencies for the "install" and "wheel [build]" steps, it is of much less help for many of the other use-cases that setuptools setup_requires facilitates. A few prominent ones:

setup.py test
setup.py pytest (and other custom commands)
setup.py check_docs

But more generally, anything that invokes setup.py for its distutils-based behaviors.

@pganssle has proposed that Setuptools should present a CLI that also supports PEP 518 for this use-case, and I'm convinced something like that is needed, as I also as hard as I try am finding it difficult to convert projects away from the tried-and-true paradigms to whole new ones without real issues arising.

But it also occurs to me that there is a project that has support for installing packages on demand and which might be a suitable bridge for setup.py, and that's this project (RWT).

RWT already builds on pip and builds throw away environments for the duration of a command (not dissimilar to what setup-requires does). And RWT was written with supplanting many setup-requires use-cases.

What RWT doesn't have is support for PEP 518 build requirements, but since it's a thin wrapper around pip, if pip were to supply an argument to pip install, namely --build-reqs, RWT could build on that also, and it would remove the need for multiple packages to have build-requirement installer support.

Here's how it would work. If invoked on a directory containing a pyproject.toml (or referencing such a file), pip would install the build requirements:

(myenv) $ pip install --build-reqs .
... pip installs build requirements defined in pyproject.toml (permanently)

But in the case where a user does not wish to install the build requirements into the environment, but does wish to invoke a setup.py command with the build requirements present, they could use rwt as the tool to do so:

$ rwt --build-reqs . -- setup.py test
... rwt invokes `install --build-reqs` into a temporary directory, adds to PYTHONPATH, then invokes setup.py test.

I was originally thinking that pip should expose a new command install-build-reqs, but then I realized that rwt only wraps pip install, so a parameter to pip install makes a lot more sense.

Alternatively, rwt could implement this support more directly (without any modifications to pip) by adding a parameter that (a) bypasses default behavior of passing parameters to pip and (b) reads the pyproject.toml and passes the parsed deps to pip install instead.

Advantages of this proposed approach:

separation of concerns (temporary, isolated install of build deps from command execution)
substantial code re-use without needing to extract a library for PEP-518
applies to many more use-cases outside of setuptools (works with other build tools)
developer does not need setuptools as a prerequisite (only pip and rwt... and only pip if rwt were to become pip run as proposed in pypa/pip#3971)
setuptools won't need to build yet another tool for installing transient requirements

Disadvantages:

Developer would need yet another tool (rwt) to gain the functionality (barring pypa/pip#3971).
rwt design doesn't keep the build environment around the way setup-requires does, so the startup cost is paid at each invocation (a cost which is substantially mitigated in some cases by pip caches).

@pganssle - if you haven't already, would you give rwt a trial run (I suggest having it installed in your global environment; it's lightweight and often immensely useful) and after let me know what you think about the proposal.

@benoit-pierre I'd be interested in your opinion also.

Rename project to pip-run

Originally, I created rwt because I wanted to have a short, pithy name. I thought rwt would be distinct and develop its own namesake and following, but it has not yet.

And since rwt doesn't convey any meaningful purpose to users, but other projects like pipenv and piptools and pip-compile have set a precedent for just prefixing the tool name with pip, I think rwt should do the same and present as pip-run. That will convey to users much more succinctly the purpose of the project and a little bit of what it does.

Environment caching

I have a number of scripts for my personal use and I found pip-run very useful for managing my dependencies. However it still takes several seconds for pip to install the packages, even if it already uses the download cache of pip.

I propose adding an option (e.g. --cache) where the directory containing the packages is not deleted after the program exits, and it can be reused as long as the dependencies list does not change. Similar tools (e.g. kotlin-main-kts) also have caching that makes rerunning a script practically free.

There are several things that need to be considered

The cached environments need to be managed, since they take up disk space (e.g. numpy takes up around 70MiB). A simple solution can be keep using /tmp and let the system manages it.
Identification of a cached environment. I think using the hash of the dependency list should work.

I think it is best to keep the functionality simple, and in the worst case just destroy and recreate the environment. But I do expect it to be able to avoid recreation of the environment if I run the same script twice.

is 'setuptools' required in requires list?

Hi @jaraco,
thanks for this great package!
I'm happily using it for bootstrapping build/test requirements in my setup.py, and replace setup_requires and tests_requre (I don't wanna compile cython into an .egg!)

Now, in the section Replacing setup_requires of your README.rst (btw, why isn't Github rendeing the ReStructuredText properly?), you recommend to add setuptools itself to the __requires__ list.

I wonder, isn't this a bit redundant?
I can run python -m rwt -- setup.py ... just fine even if I don't explicitly add setuptools to __requires__.

Isn't setuptools a dependency of pip? And since rwt in turn needs pip to do its magic, if one uses rwt, one must also have pip and hence setuptools installed -- even only to install rwt in the first place.

I could see the use of it if, say, one requires a specific minimum version of setuptools.
But why requiring a generic one?

Sorry if my question sound stupid, and thanks again for the great work :)

Unable to install packages with extra_path

Install a package like "newrelic", which specifies this in its setup.py:

        extra_path = ( "newrelic", "newrelic-%s" % package_version ),

When pip invokes setup.py install --single-version-externally-managed on the package, it generates this in the lib:

$ ls foo/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages 
newrelic-2.68.0.50  newrelic.pth

That is, the package is installed into newrelic-2.68.0.50 with a newrelic.pth file pointing at that dir. As a result, pointing PYTHONPATH at that install directory, which won't honor .pth files, won't find newrelic as a dependency.

Reading up, it appears as if extra_path is an undocumented feature.

Add comparison with pipx

In this tweet, I learned that pipx may in fact have substantial overlap with pip-run. I'd like to include reference to pipx and provide a fair comparison.

Remove in-process dependency support.

From inception, this project has exposed a .deps.on_sys_path() function that would make packages available on sys.path at run time in the current process. This technique, while an interesting proof-of-concept, hasn't seen much adoption (as far as I know). Additionally, it relies on private and deprecated aspects of setuptools (pkg_resources._initialize_master_working_set). Therefore, I'd like to remove it.

`requires` handling does not match setuptools implementation

Something like:

__requires__ = 'dependency'

is supported.

In fact, looking at the pkg_resources implementation for parse_requirements and yield_lines, something like:

__requires__ = '''
# Comment...
Babel # for translations
pyqt-distutils
PyQt5\
        >=5.8.2
setuptools-scm
setuptools>=30.3.0
'''

should work too.

9.1.0: pytest is failing in `pip_run/commands.py::pip_run.commands._separate_script` unit

I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

python3 -sBm build -w --no-isolation
because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
install .whl file in </install/prefix>
run pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>

9.0.0 was OK in exactly the same build env.
Here is pytest output:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-pip-run-9.1.0-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-pip-run-9.1.0-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.16, pytest-7.2.0, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/pip-run-9.1.0, configfile: pytest.ini
collected 31 items

pip_run/commands.py .F.                                                                                                                                              [  9%]
pip_run/deps.py ..                                                                                                                                                   [ 16%]
pip_run/read-deps.py .                                                                                                                                               [ 19%]
pip_run/scripts.py .                                                                                                                                                 [ 22%]
pip_run/text.py .                                                                                                                                                    [ 25%]
pip_run/tests/test_deps.py ......                                                                                                                                    [ 45%]
pip_run/tests/test_launch.py ..x..                                                                                                                                   [ 61%]
pip_run/tests/test_scripts.py ............                                                                                                                           [100%]

================================================================================= FAILURES =================================================================================
_______________________________________________________________ [doctest] pip_run.commands._separate_script ________________________________________________________________
011
012     Inject a double-dash before the first arg that appears to be an
013     extant Python script.
014
015     >>> _separate_script(['foo', 'bar'])
016     [['foo', 'bar'], []]
017     >>> _separate_script(['foo', 'pip-run.py', 'bar'])
Expected:
    [['foo'], ['pip-run.py', 'bar']]
Got:
    [['foo', 'pip-run.py', 'bar'], []]

/home/tkloczko/rpmbuild/BUILD/pip-run-9.1.0/pip_run/commands.py:17: DocTestFailure
============================================================================= warnings summary =============================================================================
pip_run/tests/test_scripts.py::TestNotebookDepsReader::test_one_code_block
  /home/tkloczko/rpmbuild/BUILD/pip-run-9.1.0/pip_run/scripts.py:100: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/pytest-of-tkloczko/pytest-690/test_one_code_block0/test_one_code_block.ipynb' mode='r' encoding='UTF-8'>
    doc = json.load(open(script_path))
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

pip_run/tests/test_scripts.py::TestNotebookDepsReader::test_multiple_code_blocks
  /home/tkloczko/rpmbuild/BUILD/pip-run-9.1.0/pip_run/scripts.py:100: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/pytest-of-tkloczko/pytest-690/test_multiple_code_blocks0/test_multiple_code_blocks.ipynb' mode='r' encoding='UTF-8'>
    doc = json.load(open(script_path))
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

pip_run/tests/test_scripts.py::TestNotebookDepsReader::test_code_and_markdown
  /home/tkloczko/rpmbuild/BUILD/pip-run-9.1.0/pip_run/scripts.py:100: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/pytest-of-tkloczko/pytest-690/test_code_and_markdown0/test_code_and_markdown.ipynb' mode='r' encoding='UTF-8'>
    doc = json.load(open(script_path))
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

pip_run/tests/test_scripts.py::TestNotebookDepsReader::test_jupyter_directives
  /home/tkloczko/rpmbuild/BUILD/pip-run-9.1.0/pip_run/scripts.py:100: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/pytest-of-tkloczko/pytest-690/test_jupyter_directives0/test_jupyter_directives.ipynb' mode='r' encoding='UTF-8'>
    doc = json.load(open(script_path))
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================================================================= short test summary info ==========================================================================
XFAIL pip_run/tests/test_launch.py::test_with_path_overlay - cleanup can't occur with execv; #4
FAILED pip_run/commands.py::pip_run.commands._separate_script
=========================================================== 1 failed, 29 passed, 1 xfailed, 4 warnings in 7.16s ============================================================

Here is list of installed modules in build env

Package                       Version
----------------------------- -----------------
alabaster                     0.7.12
appdirs                       1.4.4
asn1crypto                    1.5.1
attrs                         22.1.0
autocommand                   2.2.1
Babel                         2.11.0
bcrypt                        3.2.2
Brlapi                        0.8.3
build                         0.9.0
cffi                          1.15.1
charset-normalizer            3.0.1
contourpy                     1.0.6
cryptography                  38.0.4
cssselect                     1.1.0
cycler                        0.11.0
distro                        1.8.0
dnspython                     2.2.1
docutils                      0.19
exceptiongroup                1.0.0
extras                        1.0.0
fastjsonschema                2.16.1
fixtures                      4.0.0
fonttools                     4.38.0
gpg                           1.18.0-unknown
idna                          3.4
imagesize                     1.4.1
importlib-metadata            5.1.0
importlib-resources           5.10.1
iniconfig                     1.1.1
jaraco.classes                3.2.3
jaraco.packaging              9.1.1
jaraco.tidelift               1.5.0
jeepney                       0.8.0
Jinja2                        3.1.2
jsonschema                    4.17.3
jupyter_core                  5.1.0
keyring                       23.11.0
kiwisolver                    1.4.4
libcomps                      0.1.19
louis                         3.24.0
lxml                          4.9.1
MarkupSafe                    2.1.1
matplotlib                    3.6.2
more-itertools                9.0.0
nbformat                      5.7.0
numpy                         1.23.1
olefile                       0.46
packaging                     21.3
path                          16.6.0
pbr                           5.9.0
pep517                        0.13.0
Pillow                        9.3.0
pip                           22.3.1
pkgutil_resolve_name          1.3.10
platformdirs                  2.6.0
pluggy                        1.0.0
ply                           3.11
pyasn1                        0.4.8
pyasn1-modules                0.2.8
pycparser                     2.21
Pygments                      2.13.0
PyGObject                     3.42.2
pyparsing                     3.0.9
pyrsistent                    0.19.2
pytest                        7.2.0
python-dateutil               2.8.2
pytz                          2022.4
PyYAML                        6.0
requests                      2.28.1
requests-toolbelt             0.10.1
rpm                           4.17.0
rst.linker                    2.3.1
scour                         0.38.2
SecretStorage                 3.3.2
setuptools                    65.6.3
setuptools-scm                7.0.5
six                           1.16.0
snowballstemmer               2.2.0
Sphinx                        5.3.0
sphinxcontrib-applehelp       1.0.2.dev20221204
sphinxcontrib-devhelp         1.0.2.dev20221204
sphinxcontrib-htmlhelp        2.0.0
sphinxcontrib-jsmath          1.0.1.dev20221204
sphinxcontrib-qthelp          1.0.3.dev20221204
sphinxcontrib-serializinghtml 1.1.5
testtools                     2.5.0
tomli                         2.0.1
tpm2-pkcs11-tools             1.33.7
tpm2-pytss                    1.1.0
traitlets                     5.4.0
typing_extensions             4.4.0
urllib3                       1.26.12
wheel                         0.38.4
zipp                          3.11.0

9.4.0: pytest is failing in two units

I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

python3 -sBm build -w --no-isolation
because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
install .whl file in </install/prefix>
run pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>
I'm running build in build env which is cu off from access to the public network

Here is pytest output:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-pip-run-9.4.0-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-pip-run-9.4.0-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra -m 'not network'
==================================================================================== test session starts ====================================================================================
platform linux -- Python 3.8.16, pytest-7.2.0, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0, configfile: pytest.ini
collected 37 items

pip_run/commands.py ...                                                                                                                                                               [  8%]
pip_run/deps.py ..                                                                                                                                                                    [ 13%]
pip_run/persist.py .                                                                                                                                                                  [ 16%]
pip_run/read-deps.py .                                                                                                                                                                [ 18%]
pip_run/scripts.py ..                                                                                                                                                                 [ 24%]
tests/test_deps.py ..........                                                                                                                                                         [ 51%]
tests/test_launch.py ..x..                                                                                                                                                            [ 64%]
tests/test_scripts.py F..........F.                                                                                                                                                   [100%]

========================================================================================= FAILURES ==========================================================================================
_____________________________________________________________________________________ test_pkg_imported _____________________________________________________________________________________

tmp_path = PosixPath('/tmp/pytest-of-tkloczko/pytest-9/test_pkg_imported0')

    def test_pkg_imported(tmp_path):
        """
        Create a script that loads cython and ensure it runs.
        """
        jaraco.path.build(
            {
                'script': DALS(
                    """
                    import path
                    print("Successfully imported path.py")
                    """
                )
            },
            tmp_path,
        )
        script = tmp_path / 'script'
        pip_args = ['path.py']
        cmd = [sys.executable, '-m', 'pip-run'] + pip_args + ['--', str(script)]

>       out = subprocess.check_output(cmd, text=True)

tests/test_scripts.py:35:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib64/python3.8/subprocess.py:415: in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

input = None, capture_output = False, timeout = None, check = True
popenargs = (['/usr/bin/python3', '-m', 'pip-run', 'path.py', '--', '/tmp/pytest-of-tkloczko/pytest-9/test_pkg_imported0/script'],), kwargs = {'stdout': -1, 'text': True}
process = <subprocess.Popen object at 0x7fa5b25f3c10>, stdout = '', stderr = None, retcode = 1

    def run(*popenargs,
            input=None, capture_output=False, timeout=None, check=False, **kwargs):
        """Run command with arguments and return a CompletedProcess instance.

        The returned instance will have attributes args, returncode, stdout and
        stderr. By default, stdout and stderr are not captured, and those attributes
        will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them.

        If check is True and the exit code was non-zero, it raises a
        CalledProcessError. The CalledProcessError object will have the return code
        in the returncode attribute, and output & stderr attributes if those streams
        were captured.

        If timeout is given, and the process takes too long, a TimeoutExpired
        exception will be raised.

        There is an optional argument "input", allowing you to
        pass bytes or a string to the subprocess's stdin.  If you use this argument
        you may not also use the Popen constructor's "stdin" argument, as
        it will be used internally.

        By default, all communication is in bytes, and therefore any "input" should
        be bytes, and the stdout and stderr will be bytes. If in text mode, any
        "input" should be a string, and stdout and stderr will be strings decoded
        according to locale encoding, or by "encoding" if set. Text mode is
        triggered by setting any of text, encoding, errors or universal_newlines.

        The other arguments are the same as for the Popen constructor.
        """
        if input is not None:
            if kwargs.get('stdin') is not None:
                raise ValueError('stdin and input arguments may not both be used.')
            kwargs['stdin'] = PIPE

        if capture_output:
            if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
                raise ValueError('stdout and stderr arguments may not be used '
                                 'with capture_output.')
            kwargs['stdout'] = PIPE
            kwargs['stderr'] = PIPE

        with Popen(*popenargs, **kwargs) as process:
            try:
                stdout, stderr = process.communicate(input, timeout=timeout)
            except TimeoutExpired as exc:
                process.kill()
                if _mswindows:
                    # Windows accumulates the output in a single blocking
                    # read() call run on child threads, with the timeout
                    # being done in a join() on those threads.  communicate()
                    # _after_ kill() is required to collect that and add it
                    # to the exception.
                    exc.stdout, exc.stderr = process.communicate()
                else:
                    # POSIX _communicate already populated the output so
                    # far into the TimeoutExpired exception.
                    process.wait()
                raise
            except:  # Including KeyboardInterrupt, communicate handled that.
                process.kill()
                # We don't call process.wait() as .__exit__ does that for us.
                raise
            retcode = process.poll()
            if check and retcode:
>               raise CalledProcessError(retcode, process.args,
                                         output=stdout, stderr=stderr)
E               subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip-run', 'path.py', '--', '/tmp/pytest-of-tkloczko/pytest-9/test_pkg_imported0/script']' returned non-zero exit status 1.

/usr/lib64/python3.8/subprocess.py:516: CalledProcessError
----------------------------------------------------------------------------------- Captured stderr call ------------------------------------------------------------------------------------
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f5847c073a0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/path-py/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f5847c172e0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/path-py/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f5847c17550>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/path-py/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f5847c17700>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/path-py/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f5847c178b0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/path-py/
ERROR: Could not find a version that satisfies the requirement path.py (from versions: none)
ERROR: No matching distribution found for path.py
WARNING: There was an error checking the latest version of pip.
Traceback (most recent call last):
  File "/usr/lib64/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0/pip-run.py", line 4, in <module>
    __name__ == '__main__' and run()
  File "/home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0/pip_run/__init__.py", line 18, in run
    with deps.load(*deps.not_installed(pip_args)) as home:
  File "/usr/lib64/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0/pip_run/deps.py", line 74, in load
    Install.parse(args) and empty(target) and subprocess.check_call(cmd, env=env)
  File "/usr/lib64/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('/usr/bin/python3', '-m', 'pip', 'install', '-t', PosixPath('/tmp/pip-run-2owvtu_x'), 'path.py')' returned non-zero exit status 1.
___________________________________________________________________________ test_pkg_loaded_from_alternate_index ____________________________________________________________________________

tmp_path = PosixPath('/tmp/pytest-of-tkloczko/pytest-9/test_pkg_loaded_from_alternate0')

    def test_pkg_loaded_from_alternate_index(tmp_path):
        """
        Create a script that loads cython from an alternate index
        and ensure it runs.
        """
        jaraco.path.build(
            {
                'script': DALS(
                    """
                    __requires__ = ['path.py']
                    __index_url__ = 'https://devpi.net/root/pypi/+simple/'
                    import path
                    print("Successfully imported path.py")
                    """
                )
            },
            tmp_path,
        )
        cmd = [sys.executable, '-m', 'pip-run', '-v', '--', str(tmp_path / 'script')]

>       out = subprocess.check_output(cmd, text=True)

tests/test_scripts.py:176:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib64/python3.8/subprocess.py:415: in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

input = None, capture_output = False, timeout = None, check = True
popenargs = (['/usr/bin/python3', '-m', 'pip-run', '-v', '--', '/tmp/pytest-of-tkloczko/pytest-9/test_pkg_loaded_from_alternate0/script'],), kwargs = {'stdout': -1, 'text': True}
process = <subprocess.Popen object at 0x7fa5b25373a0>, stdout = 'Looking in indexes: https://devpi.net/root/pypi/+simple/\n', stderr = None, retcode = 1

    def run(*popenargs,
            input=None, capture_output=False, timeout=None, check=False, **kwargs):
        """Run command with arguments and return a CompletedProcess instance.

        The returned instance will have attributes args, returncode, stdout and
        stderr. By default, stdout and stderr are not captured, and those attributes
        will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them.

        If check is True and the exit code was non-zero, it raises a
        CalledProcessError. The CalledProcessError object will have the return code
        in the returncode attribute, and output & stderr attributes if those streams
        were captured.

        If timeout is given, and the process takes too long, a TimeoutExpired
        exception will be raised.

        There is an optional argument "input", allowing you to
        pass bytes or a string to the subprocess's stdin.  If you use this argument
        you may not also use the Popen constructor's "stdin" argument, as
        it will be used internally.

        By default, all communication is in bytes, and therefore any "input" should
        be bytes, and the stdout and stderr will be bytes. If in text mode, any
        "input" should be a string, and stdout and stderr will be strings decoded
        according to locale encoding, or by "encoding" if set. Text mode is
        triggered by setting any of text, encoding, errors or universal_newlines.

        The other arguments are the same as for the Popen constructor.
        """
        if input is not None:
            if kwargs.get('stdin') is not None:
                raise ValueError('stdin and input arguments may not both be used.')
            kwargs['stdin'] = PIPE

        if capture_output:
            if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
                raise ValueError('stdout and stderr arguments may not be used '
                                 'with capture_output.')
            kwargs['stdout'] = PIPE
            kwargs['stderr'] = PIPE

        with Popen(*popenargs, **kwargs) as process:
            try:
                stdout, stderr = process.communicate(input, timeout=timeout)
            except TimeoutExpired as exc:
                process.kill()
                if _mswindows:
                    # Windows accumulates the output in a single blocking
                    # read() call run on child threads, with the timeout
                    # being done in a join() on those threads.  communicate()
                    # _after_ kill() is required to collect that and add it
                    # to the exception.
                    exc.stdout, exc.stderr = process.communicate()
                else:
                    # POSIX _communicate already populated the output so
                    # far into the TimeoutExpired exception.
                    process.wait()
                raise
            except:  # Including KeyboardInterrupt, communicate handled that.
                process.kill()
                # We don't call process.wait() as .__exit__ does that for us.
                raise
            retcode = process.poll()
            if check and retcode:
>               raise CalledProcessError(retcode, process.args,
                                         output=stdout, stderr=stderr)
E               subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip-run', '-v', '--', '/tmp/pytest-of-tkloczko/pytest-9/test_pkg_loaded_from_alternate0/script']' returned non-zero exit status 1.

/usr/lib64/python3.8/subprocess.py:516: CalledProcessError
----------------------------------------------------------------------------------- Captured stderr call ------------------------------------------------------------------------------------
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6ac61e1280>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /root/pypi/+simple/path-py/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6ac61e1580>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /root/pypi/+simple/path-py/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6ac61e1730>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /root/pypi/+simple/path-py/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6ac61e18e0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /root/pypi/+simple/path-py/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6ac61e1a90>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /root/pypi/+simple/path-py/
ERROR: Could not find a version that satisfies the requirement path.py (from versions: none)
ERROR: No matching distribution found for path.py
WARNING: There was an error checking the latest version of pip.
Traceback (most recent call last):
  File "/usr/lib64/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0/pip-run.py", line 4, in <module>
    __name__ == '__main__' and run()
  File "/home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0/pip_run/__init__.py", line 18, in run
    with deps.load(*deps.not_installed(pip_args)) as home:
  File "/usr/lib64/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0/pip_run/deps.py", line 74, in load
    Install.parse(args) and empty(target) and subprocess.check_call(cmd, env=env)
  File "/usr/lib64/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('/usr/bin/python3', '-m', 'pip', 'install', '-t', PosixPath('/tmp/pip-run-xz8c_iz3'), '-v', '--index-url', 'https://devpi.net/root/pypi/+simple/', 'path.py')' returned non-zero exit status 1.
================================================================================== short test summary info ==================================================================================
XFAIL tests/test_launch.py::test_with_path_overlay - cleanup can't occur with execv; #4
FAILED tests/test_scripts.py::test_pkg_imported - subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip-run', 'path.py', '--', '/tmp/pytest-of-tkloczko/pytest-9/test_pkg_imported0/script']' returned non-zero exit sta...
FAILED tests/test_scripts.py::test_pkg_loaded_from_alternate_index - subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip-run', '-v', '--', '/tmp/pytest-of-tkloczko/pytest-9/test_pkg_loaded_from_alternate0/script']' returned non-zero ...
=================================================================== 2 failed, 34 passed, 1 xfailed in 1647.11s (0:27:27) ====================================================================

Here is list of installed modules in build env

Package                       Version
----------------------------- -----------------
alabaster                     0.7.12
appdirs                       1.4.4
asn1crypto                    1.5.1
attrs                         22.2.0
autocommand                   2.2.1
Babel                         2.11.0
bcrypt                        3.2.2
Brlapi                        0.8.3
build                         0.9.0
cffi                          1.15.1
charset-normalizer            3.0.1
contourpy                     1.0.6
cryptography                  38.0.4
cssselect                     1.1.0
cycler                        0.11.0
distro                        1.8.0
dnspython                     2.2.1
docutils                      0.19
exceptiongroup                1.0.0
extras                        1.0.0
fastjsonschema                2.16.1
fixtures                      4.0.0
fonttools                     4.38.0
gpg                           1.18.0-unknown
idna                          3.4
imagesize                     1.4.1
importlib-metadata            5.1.0
importlib-resources           5.10.1
inflect                       5.0.0
iniconfig                     1.1.1
jaraco.classes                3.2.3
jaraco.context                4.2.0
jaraco.functools              3.5.2
jaraco.packaging              9.1.1
jaraco.path                   3.4.0
jaraco.text                   3.11.0
jaraco.tidelift               1.5.0
jeepney                       0.8.0
Jinja2                        3.1.2
jsonschema                    4.17.3
jupyter_core                  5.1.1
keyring                       23.11.0
kiwisolver                    1.4.4
libcomps                      0.1.19
louis                         3.24.0
lxml                          4.9.1
MarkupSafe                    2.1.1
matplotlib                    3.6.2
more-itertools                9.0.0
nbformat                      5.7.0
numpy                         1.23.1
olefile                       0.46
packaging                     21.3
path                          16.6.0
pbr                           5.9.0
pep517                        0.13.0
Pillow                        9.3.0
pip                           22.3.1
pkgutil_resolve_name          1.3.10
platformdirs                  2.6.0
pluggy                        1.0.0
ply                           3.11
pyasn1                        0.4.8
pyasn1-modules                0.2.8
pycparser                     2.21
Pygments                      2.13.0
PyGObject                     3.42.2
pyparsing                     3.0.9
pyrsistent                    0.19.2
pytest                        7.2.0
python-dateutil               2.8.2
pytz                          2022.4
PyYAML                        6.0
requests                      2.28.1
requests-toolbelt             0.10.1
rpm                           4.17.0
rst.linker                    2.3.1
scour                         0.38.2
SecretStorage                 3.3.2
setuptools                    65.6.3
setuptools-scm                7.0.5
six                           1.16.0
snowballstemmer               2.2.0
Sphinx                        5.3.0
sphinxcontrib-applehelp       1.0.2.dev20221204
sphinxcontrib-devhelp         1.0.2.dev20221204
sphinxcontrib-htmlhelp        2.0.0
sphinxcontrib-jsmath          1.0.1.dev20221204
sphinxcontrib-qthelp          1.0.3.dev20221204
sphinxcontrib-serializinghtml 1.1.5
testtools                     2.5.0
tomli                         2.0.1
tpm2-pkcs11-tools             1.33.7
tpm2-pytss                    1.1.0
traitlets                     5.8.0
typing_extensions             4.4.0
urllib3                       1.26.12
wheel                         0.38.4
zipp                          3.11.0

Automatic dependency management

Hey @jaraco. I am reading about the project and for some reason I want to believe that rwt is able to figure out and fetch dependencies automatically, but so far I haven't found proof of that - we still need to create a custom "import" section just for rwt. In this respect rwt is still manual dependency management like http://docs.groovy-lang.org/latest/html/documentation/grape.html

Fails on homebrew Python

Due to pypa/pip#4106, pip install --target doesn't work on Homebrew Python (or other Pythons that set prefix). As a result, rwt is nonviable in those environments.

In this comment, silverwind provides a workaround. Perhaps rwt can employ that workaround.

Add support for other PIP parameters in script inference

One key feature of rwt is that it can be readily used on a single script to resolve dependencies declared in the script and install them, using the setuptools convention of __requires__. This convention, however, does not allow for other options to install, such as the ever-important --index-url. There should be another, comparable mechanism for a script to declare other parameters to pip.

Unable to pip output from one pip-run to another pip-run

Consider the use-case where you want to test the encoding/decoding of some string against two versions of a library. You could do two invocations of pip-run:

$ pip-run -q itsdangerous==0.24 -- -c "import itsdangerous; print(itsdangerous.Signer(b'secret-key').sign(b'my string').decode('ascii'))"
my string.wh6tMHxLgJqB6oY1uT73iMlyrOA
$ echo 'my string.wh6tMHxLgJqB6oY1uT73iMlyrOA' | pip-run -q itsdangerous==1.1 -- -c "import itsdangerous, sys; print(itsdangerous.Signer('secret-key').unsign(sys.stdin.read()))"
...

And that works fine, but anyone who has spent more than 30 seconds in Unix would expect to be able to take the output of the first command and pipe it into the second command to avoid having to emit and then re-enter the intermediate result. However, if you do this with pip-run, it fails with a BrokenPipeError:

~ $ pip-run -q itsdangerous==0.24 -- -c "import itsdangerous; print(itsdangerous.Signer(b'secret-key').sign(b'my string').decode('ascii'))" | pip-run -q itsdangerous==1.1 -- -c "import itsdangerous, sys; print(itsdangerous.Signer('secret-key').unsign(sys.stdin.read()))"
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/bin/pip-run", line 11, in <module>
    sys.exit(run())
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pip_run/__init__.py", line 18, in run
    with deps.load(*deps.not_installed(pip_args)) as home:
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pip_run/deps.py", line 74, in load
    with _patch_prefix():
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pip_run/deps.py", line 89, in _patch_prefix
    with _save_file(cfg_fn):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pip_run/deps.py", line 104, in _save_file
    raise NotImplementedError(tmpl.format(**locals()))
NotImplementedError: Unsupported with extant /Users/jaraco/.pydistutils.cfg
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe

This all would work except for the (undocumented) caveat that pip-run relies on ~/.pydistutils.cfg to inject certain functionality... and so two invocations of pip-run cannot run simultaneously for the same user.

Here is the code that invokes this behavior, which implicates #14 and pypa/pip#4106.

This issue only affects invocations where the pip install commands from pip-run overlap.

Interrupt causes traceback in rwt stack

When sending a sigint to the command, the underlying command will exit normally, but then rwt will receive the KeyboardInterrupt as well. Where possible, rwt should probably just use execv to replace the process.

app_paths dependency is a nuisance

Per 3463926#r94301964, @mgorny reports that the app_paths dependency is problemmatic. Let's explore ways to fix that.

implement limited requirements support in comments

Let's implement option 4 from #44 but with tweaks as proposed in pypa/pipx#913 as an additional supported format for specifying requirements in a script.

Honor .pth files in target directory

Currently, .pth files are only honored on Python 3.2 and earlier due to this workaround.

Python 3.3 and later are excused because the only perceived need for .pth files at the time of the implementation was to support namespace packages, packages which work fine in PEP 420 mode naturally on Python 3.3 and later.

However, I've since learned of other packages such as future-fstrings that install .pth files to enable behavior on startup.

This leads me to believe rwt should no longer exclude later Python versions from the .pth file support... although this increases the impact of the issue that all rwt invocations will now mask any existing sitecustomize.py.

Remove .pydistutils.cfg workaround

In #14, a workaround was added to support homebrew and other environments that override the prefix. In pypa/pip#6008, a fix was merged to pip to support --target in those environments and released with pip 19.2. That means this project can drop that workaround as long as it requires pip 19.2 or later.

pip 10 errors on no requirements

In Pip 9.x and earlier, pip install with no requirements would issue a warning. In pip 10, that's an error. Since rwt filters out packages already satisfied, it may end up with an empty set, triggering this warning. If no requirements are indicated, it should avoid the pip install invocation and the emergent error.

Error running script with f-strings on older Pythons

You can use future-fstrings to run a script with fstrings on older Pythons:

rwt master $ cat test-fstrings.py
# coding: future_fstrings

__requires__ = ['future-fstrings']

print(f'This requires {__requires__}')
rwt master $ python3.5 -m rwt -q future-fstrings -- test-fstrings.py
This requires ['future-fstrings']

But it only works if future-fstrings was specified on the command line. It won't be inferred from the __requires__ because the ast.parse cannot handle the invalid Syntax during the DepsReader._read call.

I think it would be useful if f strings weren't an impediment to the DepsReader.

Give pip_run.read-deps an option for newline-terminated output

Feature request: Give python -m pip_run.read-deps a --newline (or whatever you want to call it) option that would cause the individual requirements to be output on separate lines. This would make it easy to create requirements.txt files for __requires__-using scripts, especially when dealing with requirements with markers like importlib-metadata>=3.6; python_version < "3.10", which are currently output with spaces embedded, making it difficult to separate them from adjacent requirements.

No help available?

I tried py -m rwt --help to get basic usage information, and I appear to get pip's help. rwt should support a standard help invocation.

No module named _vendor

When invoking rwt on debian, the command fails thus:

$ python -m rwt requests
/usr/bin/python: No module named _vendor; 'rwt' is a package and cannot be directly executed

The issue is almost certainly the from pip._vendor import pkg_resources in deps.py.

New `with_path_overlay` launcher doesn't clean up

Q: When does a with context never exit?
A: When the command in the context invokes execv.

It occurred to me today that using execve to launch the child process would eliminate the cleanup logic for the temporary directory.

I started to try to think of some clever way to cause the cleanup logic to be invoked after the child process has exited:

somehow inject an atexit behavior into the child Python process.
run a separate subprocess that monitors the parent and cleans up when the parent exits.
invent an OS-level process-temporary store, a part of the file system that the OS will guarantee will be cleaned up when the process exits. How cool would that be? I'm surprised such a thing doesn't already exist.

Then I realized, I'm trying too hard and I should probably just switch rwt back to the old subprocess behavior.

double-dash is irritating, especially for script invocation

In pypa/pip#3971 (comment), @pfmoore has to say:

I find having to include the -- to separate the command parameters clumsy, and I'm not in love with details like the default being to print all of pip's output. Which all adds up to a general sense of "I never think to use it, because it's sort of alright, but irritating". Sorry, I know that's annoyingly vague, and I do think that the UI is the hardest part of this to get right. What I want is to be able to type pip-run my_script.py arg1 arg2 arg3, and have it run my script with all dependencies available. The other features of pip-run are secondary (to me).

In this issue, I'll focus on the double-dash behavior.

The reason for the -- is to clearly separate the parameters to the script from parameters to pip/pip-run. Since there are two concerns to deal with here, there needs to be clear separation. For me, 95% of the usage is on the install parameters side, with the right hand side accepting any argument to Python (sometimes no args (for a repl), sometimes -c, sometimes -m, and sometimes a script). This generalization means the tool has a wide variety of applications.

I'd like to improve the reported scenario, but I struggle a bit with how to do that without drastically degrading the current experience.

One possible improvement could be to allow the -- to be optional in the specific case where a script is the first parameter. That is, if the first parameter is an extant file, that's not a valid parameter to pip install, so pip-run could infer a -- before it. If someone wanted to pass arguments to the interpreter or to pip, they would need to bring back the --, but for convenience, a simple script invocation could omit it. WDYT?

Namespace packages don't work on older Pythons

Due to the way pip installs namespace packages, relying on .pth files to satisfy namespace packages, and because .pth files are only honored in site-packages, rwt cannot load packages that are namespace packages on older Pythons like 2.7.

$ python3.5 -m rwt backports.functools_lru_cache -- -c "import backports"
Loading requirements using backports.functools_lru_cache
$ python2.7 -m rwt backports.functools_lru_cache -- -c "import backports"
Loading requirements using backports.functools_lru_cache
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named backports

rwt fails to install local checkouts

Since the fix for #28, attempting to install a package from a local checkout results in nothing being installed:

$ rwt ~/p/twine
Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 26 2018, 23:26:24)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import twine
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'twine'

PYTHONPATH should be augmented, not replaced

Invocation should pass the subprocess result code

When invoking the Python subprocess, the exit code should be relayed back to the caller. For example:

$ python -m rwt -- -c "raise SystemExit(1)"
Loading requirements using 
$ echo $?
0

Instead, 1 should be emitted.

Add a console entry point

I gave this a try, and my immediate thought was to run it as "rwt". I know the docs say to use "python -m rwt", but maybe add a console entry point as a convenience?

Fails when requirements satisfied and parameters passed

Consider:

~ $ rwt -q inflect
ERROR: You must give at least one requirement to install (see "pip help install")
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/bin/rwt", line 11, in <module>
    sys.exit(run())
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/rwt/__init__.py", line 15, in run
    with deps.load(*deps.not_installed(pip_args)) as home:
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/rwt/deps.py", line 49, in load
    args and subprocess.check_call(cmd)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 328, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('/Library/Frameworks/Python.framework/Versions/3.7/bin/python3', '-m', 'pip', 'install', '-t', '/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/rwt-dwyx_rwx', '-q')' returned non-zero exit status 1.

In #25, we addressed this issue for simple declarations to pip, but it seems not to be working when other parameters like -q are present.

Git URLs no longer supported in requires?

Somewhere between versions 2.15.1 and 3.0, rwt stopped supporting (specifically, it appears to flat-out ignore) package specifiers of the form git+ssh://[email protected]/repo.git in __requires__ lists, though it continues to support them in requirements.txt files. I see no mention of this change in CHANGES.rst; was it intentional? If not, could we have the old behavior back?

[upstream] Better support for already-installed requirements

I have a scenario where if I make the following requirements.txt file:

cryptography<=3.2.1,>2.9.2
azure-storage-blob==12.8.1

and pass it with -r, everything is fine.

If I add this to the script i'm running, and remove the -r flag:

__requires__ = [
    'cryptography<=3.2.1,>2.9.2',
    'azure-storage-blob>12',
]

I get an error because my site-packages has jwt 1.1.0 and cryptography 3.2.1 installed:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
jwt 1.1.0 requires cryptography<=3.2.1,>2.9.2, but you have cryptography 3.4.7 which is incompatible.

What i'd like:

a flag that will give me the -r behavior without having to make another separate file

What I'd love:

a flag that will make it not matter what I have installed in site-packages at all. I just want a one line command that will run my script regardless of what might have been installed with pip on the machine in the past.

ResourceWarning in PathReader

When attempting to use pip-run in Setuptools tests, which turn warnings into errors, I see this error:

E               pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
E               
E               Traceback (most recent call last):
E                 File "/Users/jaraco/code/public/pypa/setuptools/.tox/python/lib/python3.10/site-packages/pip_run/launch.py", line 64, in _build_env
E                   joined = os.pathsep.join(items)
E               ResourceWarning: unclosed file <_io.TextIOWrapper name='/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-71fkg9i3/distutils-precedence.pth' mode='r' encoding='UTF-8'>

.tox/python/lib/python3.10/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning

It seems that Python warns when using open without its context manager, such as here.

Namespace packages fail when namespace is installed

Distinct from #1, where the namespace package isn't recognized at all, there's another issue that affects all versions of Python.

$ python -m rwt backports.functools_lru_cache -- -c "import backports.functools_lru_cache"
Loading requirements using backports.functools_lru_cache
$ python -c "import backports"                                                          
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named 'backports'
$ python -m pip install backports.unittest_mock
Collecting backports.unittest_mock
  Downloading backports.unittest_mock-1.1.1-py2.py3-none-any.whl
Installing collected packages: backports.unittest-mock
Successfully installed backports.unittest-mock-1.1.1
$ python -c "import backports"
$ python -m rwt backports.functools_lru_cache -- -c "import backports.functools_lru_cache"
Loading requirements using backports.functools_lru_cache
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named 'backports.functools_lru_cache'

The problem is that the -nspkg.pth handler is interfering with the PEP-420 loader. Remove it and both packages import nicely.

$ rm /python/lib/python3.5/site-packages/backports.unittest_mock-1.1.1-py3.5-nspkg.pth
$ python -m rwt backports.functools_lru_cache -- -c "import backports.functools_lru_cache; import backports.unittest_mock"
Loading requirements using backports.functools_lru_cache

PIP 10 warns when scripts aren't on PATH

See pypa/pip#4553.

I was thinking we could suppress this error, but I noticed in pypa/pip#5200 that the warning isn't suppressed with an environment variable.

setup.py containing requires fails when run without rwt

hi @jaraco,

Here's a setup.py which defines __requires__ for use with rwt tool. It's taken straight from your README.rst for the sake of example.

#!/usr/bin/env python

__requires__ = ['setuptools', 'setuptools_scm']

from setuptools import setup

setup(
    name='test_rwt',
    use_scm_version=True,
    setup_requires=__requires__,
)

Now, if I try to run this without also invoking python -m rwt -- setup.py ..., like the usual:

$ python3 setup.py --help

I get this error from pkg_resources:

Traceback (most recent call last):
  File "setup_rwt.py", line 5, in <module>
    from setuptools import setup
  File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/setuptools/__init__.py", line 12, in <module>
    import setuptools.version
  File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/setuptools/version.py", line 1, in <module>
    import pkg_resources
  File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3018, in <module>
    @_call_aside
  File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3002, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3031, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 654, in _build_master
    ws.require(__requires__)
  File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 962, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 848, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'setuptools_scm' distribution was not found and is required by the application

The error happens when a requirement listed in __requires__ (like setuptools_scm here) is not already installed in the current environment.

It seems like pkg_resources (imported by setuptools) also attempts to use the __requires__ variable, in WorkinSet._build_master method:

https://github.com/pypa/setuptools/blob/089cdeb489a0fa94d11b7307b54210ef9aa40511/pkg_resources/__init__.py#L640-L658

This behavior is documented in the pkg_resources docs:
http://setuptools.readthedocs.io/en/latest/pkg_resources.html#workingset-objects

All distributions available directly on sys.path will be activated automatically when pkg_resources is imported. This behaviour can cause version conflicts for applications which require non-default versions of those distributions. To handle this situation, pkg_resources checks for a requires attribute in the main module when initializing the default working set, and uses this to ensure a suitable version of each affected distribution is activated. For example:

__requires__ = ["CherryPy < 3"] # Must be set before pkg_resources import
import pkg_resources

In your REAME.rst, you say that this technique:

When invoked with rwt, the dependencies will be assured before the script is run, or if run with setuptools, the dependencies will be loaded using the older technique, so the script is backward compatible.

However, it appears to me that one cannot have such a backward-compatible setup.py which also takes advantage of rwt via the __requires__ variable, because once one uses that, it fails as one tries to run it in the old-fashioned way.

Am I missing something?
Thank you for your help.

In the meantime I think I'll just use python3 -m rwt -r requirements/setup.txt -- setup.py ... instead of __requires__.

Drop support for `__dependency_links__`

In #24, this project adopted support for specifying --dependency-links in a script to support VCS installation. Since then (or perhaps before), pip added support for VCS URLs directly in requirements. Let's remove this now deprecated functionality.

PYTHONPATH leaks into subprocesses

The way rwt works, it sets PYTHONPATH when invoking its child Python process, to present the new packages to it. And if that Python process invokes subprocesses, they inherit the environment by default. Sometimes, this inheritance is useful, because especially if the subprocess is the same Python (sys.executable). Other times, it's an impediment.

Consider the case where rwt is invoked on Python 3 and loads a number of libraries, but then the subprocess invokes a Python 2 subprocess in another environment. Now that Python 2 subprocess finds itself trying to import modules that were installed and configured for Python 3, and hilarity ensues.

packaging dependency missing

This commit added the line import packaging.requirements to the code, but packaging was not added as a dependency for the project. Trying to use pip-run in a fresh environment thus fails with ModuleNotFoundError: No module named 'packaging'.

Shared redesign of embedded requirements feature

Background

In pypa/pip#3971, key members of the PyPA have leveled critiques of the embedded requirements feature, where a user can supply requirements and other common installer directives to signal to the tool how (best) to run the script. This feature emerged from the use-case where a script author would like to distribute a single file and have that file be runnable by any number of end users with minimal guidance, allowing the file to be published to a gist or alongside other scripts in a directory without needing additional context for executability.

Goals

the tool should be able to infer install requirements and index URL.
the tool must be able to parse these directives without executing the script (as executing the script should be able to rely on the presence of those items).
the syntax should be as intuitive as possible. As a corollary, the syntax should aim to re-use syntax familiar to the user (primarily the author, but also the end user).
(optional) the tool should be able to infer additional installer directives such as --extra-index-url or --quiet.

Remove dependency on pkg_resources

The code currently has an implicit dependency on pkg_resources, which is satisfied by its dependency on pip (which vendors that pkg_resources). It would be nice to remove this dependency. By addressing #39, the only remaining usage of pkg_resources is on determining which package versions are already satisfied. I considered using importlib_metadata to do that, but ran into python-devs/importlib_metadata#82.

pip_run.read-deps fails with "type object 'DepsReader' has no attribute 'load'"

Currently, as of v8.2.0, trying to run python3 -m pip_run.read-deps on a script with __requires__ fails with the following error:

Traceback (most recent call last):
  File "/usr/local/Cellar/[email protected]/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/Cellar/[email protected]/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.9/site-packages/pip_run/read-deps.py", line 16, in <module>
    __name__ == '__main__' and run()
  File "/usr/local/lib/python3.9/site-packages/pip_run/read-deps.py", line 12, in run
    deps = DepsReader.load(script).read()
AttributeError: type object 'DepsReader' has no attribute 'load'

This failure appears to be due to read-deps.py not having been updated after DepsReader.load was split out into SourceDepsReader in commit f943b56.