jaraco / pip-run Goto Github PK
View Code? Open in Web Editor NEWpip-run - dynamic dependency loader for Python
License: MIT License
pip-run - dynamic dependency loader for Python
License: MIT License
This commit added the line import packaging.requirements
to the code, but packaging
was not added as a dependency for the project. Trying to use pip-run in a fresh environment thus fails with ModuleNotFoundError: No module named 'packaging'
.
Consider the use-case where you want to test the encoding/decoding of some string against two versions of a library. You could do two invocations of pip-run:
$ pip-run -q itsdangerous==0.24 -- -c "import itsdangerous; print(itsdangerous.Signer(b'secret-key').sign(b'my string').decode('ascii'))"
my string.wh6tMHxLgJqB6oY1uT73iMlyrOA
$ echo 'my string.wh6tMHxLgJqB6oY1uT73iMlyrOA' | pip-run -q itsdangerous==1.1 -- -c "import itsdangerous, sys; print(itsdangerous.Signer('secret-key').unsign(sys.stdin.read()))"
...
And that works fine, but anyone who has spent more than 30 seconds in Unix would expect to be able to take the output of the first command and pipe it into the second command to avoid having to emit and then re-enter the intermediate result. However, if you do this with pip-run, it fails with a BrokenPipeError:
~ $ pip-run -q itsdangerous==0.24 -- -c "import itsdangerous; print(itsdangerous.Signer(b'secret-key').sign(b'my string').decode('ascii'))" | pip-run -q itsdangerous==1.1 -- -c "import itsdangerous, sys; print(itsdangerous.Signer('secret-key').unsign(sys.stdin.read()))"
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/bin/pip-run", line 11, in <module>
sys.exit(run())
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pip_run/__init__.py", line 18, in run
with deps.load(*deps.not_installed(pip_args)) as home:
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 112, in __enter__
return next(self.gen)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pip_run/deps.py", line 74, in load
with _patch_prefix():
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 112, in __enter__
return next(self.gen)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pip_run/deps.py", line 89, in _patch_prefix
with _save_file(cfg_fn):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 112, in __enter__
return next(self.gen)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pip_run/deps.py", line 104, in _save_file
raise NotImplementedError(tmpl.format(**locals()))
NotImplementedError: Unsupported with extant /Users/jaraco/.pydistutils.cfg
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe
This all would work except for the (undocumented) caveat that pip-run relies on ~/.pydistutils.cfg to inject certain functionality... and so two invocations of pip-run cannot run simultaneously for the same user.
Here is the code that invokes this behavior, which implicates #14 and pypa/pip#4106.
This issue only affects invocations where the pip install
commands from pip-run
overlap.
See pypa/pip#4553.
I was thinking we could suppress this error, but I noticed in pypa/pip#5200 that the warning isn't suppressed with an environment variable.
On Ubuntu Xenial with sudo aptitude install -y python3-pip
, and python3 -m pip install --user rwt
:
jaraco@dev-jaraco:~$ python3 -m rwt pymongo
Loading requirements using pymongo
Exception:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 209, in main
status = self.run(options, args)
File "/usr/lib/python3/dist-packages/pip/commands/install.py", line 335, in run
prefix=options.prefix_path,
File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 732, in install
**kwargs
File "/usr/lib/python3/dist-packages/pip/req/req_install.py", line 837, in install
self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
File "/usr/lib/python3/dist-packages/pip/req/req_install.py", line 1039, in move_wheel_files
isolated=self.isolated,
File "/usr/lib/python3/dist-packages/pip/wheel.py", line 247, in move_wheel_files
prefix=prefix,
File "/usr/lib/python3/dist-packages/pip/locations.py", line 153, in distutils_scheme
i.finalize_options()
File "/usr/lib/python3.5/distutils/command/install.py", line 273, in finalize_options
raise DistutilsOptionError("can't combine user with prefix, "
distutils.errors.DistutilsOptionError: can't combine user with prefix, exec_prefix/home, or install_(plat)base
Traceback (most recent call last):
File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/jaraco/.local/lib/python3.5/site-packages/rwt/__main__.py", line 12, in <module>
with deps.load(*pip_args) as home:
File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
return next(self.gen)
File "/home/jaraco/.local/lib/python3.5/site-packages/rwt/deps.py", line 42, in load
subprocess.check_call(cmd)
File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('/usr/bin/python3', '-m', 'pip', 'install', '-q', '-t', '/tmp/rwt-sp95m3xg', 'pymongo')' returned non-zero exit status 2
You can use future-fstrings to run a script with fstrings on older Pythons:
rwt master $ cat test-fstrings.py
# coding: future_fstrings
__requires__ = ['future-fstrings']
print(f'This requires {__requires__}')
rwt master $ python3.5 -m rwt -q future-fstrings -- test-fstrings.py
This requires ['future-fstrings']
But it only works if future-fstrings was specified on the command line. It won't be inferred from the __requires__
because the ast.parse
cannot handle the invalid Syntax during the DepsReader._read
call.
I think it would be useful if f strings weren't an impediment to the DepsReader.
Let's implement option 4 from #44 but with tweaks as proposed in pypa/pipx#913 as an additional supported format for specifying requirements in a script.
hi @jaraco,
Here's a setup.py which defines __requires__
for use with rwt
tool. It's taken straight from your README.rst for the sake of example.
#!/usr/bin/env python
__requires__ = ['setuptools', 'setuptools_scm']
from setuptools import setup
setup(
name='test_rwt',
use_scm_version=True,
setup_requires=__requires__,
)
Now, if I try to run this without also invoking python -m rwt -- setup.py ...
, like the usual:
$ python3 setup.py --help
I get this error from pkg_resources:
Traceback (most recent call last):
File "setup_rwt.py", line 5, in <module>
from setuptools import setup
File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/setuptools/__init__.py", line 12, in <module>
import setuptools.version
File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/setuptools/version.py", line 1, in <module>
import pkg_resources
File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3018, in <module>
@_call_aside
File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3002, in _call_aside
f(*args, **kwargs)
File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3031, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 654, in _build_master
ws.require(__requires__)
File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 962, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/local/var/pyenv/versions/3.6.0/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 848, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'setuptools_scm' distribution was not found and is required by the application
The error happens when a requirement listed in __requires__
(like setuptools_scm
here) is not already installed in the current environment.
It seems like pkg_resources
(imported by setuptools) also attempts to use the __requires__
variable, in WorkinSet._build_master
method:
This behavior is documented in the pkg_resources docs:
http://setuptools.readthedocs.io/en/latest/pkg_resources.html#workingset-objects
All distributions available directly on sys.path will be activated automatically when pkg_resources is imported. This behaviour can cause version conflicts for applications which require non-default versions of those distributions. To handle this situation, pkg_resources checks for a requires attribute in the main module when initializing the default working set, and uses this to ensure a suitable version of each affected distribution is activated. For example:
__requires__ = ["CherryPy < 3"] # Must be set before pkg_resources import
import pkg_resources
In your REAME.rst, you say that this technique:
When invoked with rwt, the dependencies will be assured before the script is run, or if run with setuptools, the dependencies will be loaded using the older technique, so the script is backward compatible.
However, it appears to me that one cannot have such a backward-compatible setup.py which also takes advantage of rwt
via the __requires__
variable, because once one uses that, it fails as one tries to run it in the old-fashioned way.
Am I missing something?
Thank you for your help.
In the meantime I think I'll just use python3 -m rwt -r requirements/setup.txt -- setup.py ...
instead of __requires__
.
From inception, this project has exposed a .deps.on_sys_path()
function that would make packages available on sys.path
at run time in the current process. This technique, while an interesting proof-of-concept, hasn't seen much adoption (as far as I know). Additionally, it relies on private and deprecated aspects of setuptools (pkg_resources._initialize_master_working_set
). Therefore, I'd like to remove it.
In pypa/pip#3971 (comment), @pfmoore characterizes the issue:
I'm not in love with details like the default being to print all of pip's output, [contributing] to a general sense of "I never think to use it, because it's sort of alright, but irritating". What I want is to be able to type
pip-run my_script.py arg1 arg2 arg3
, and have it run my script with all dependencies available. The other features ofpip-run
are secondary (to me).
It is often super-useful to have the output emitted, because it provides a report of what was installed. This is great when copy/pasting for a bug report, because it provides a snapshot of the relevant package versions (and method of installation).
I've also bemoaned the fact that the output includes pip installer output by default. I'd very much prefer to suppress that output by default, but make it an option to re-enable.
Because that's not the case, I often invoke pip-run with -q
to silence that behavior (which adds another annoyance to the experience, similar to #57. There's no clean, quiet experience.
The reason the behavior is because that's the default pip install behavior and pip-run is trying not to re-invent the interface but instead loosely couple with the existing interface. In that way, many other installer options are readily available, such as -v
if you need more detail about why the install is failing, or --index
or --use-pep517
. Without that generalization, it would be a maintenance challenge (nightmare?) to keep up with the mapping between the pip-run UX and the pip install UX.
I'd be interested in making the install 'quiet by default', but I'm not sure how to do that safely and still honor the flags to pip install.
For instance, if pip-run were to set PIP_QUIET=1, does that have the intended effect? Can the user override that with -v
?
I ran some quick tests and at first blush, it looks like everything works as one might expect, with the installer details being suppressed by default, but re-activated with -v
, and -vvv
has the same effect as -vv
without PIP_QUIET.
So I think this one has an easy fix - set PIP_QUIET when invoking pip install.
Currently, as of v8.2.0, trying to run python3 -m pip_run.read-deps
on a script with __requires__
fails with the following error:
Traceback (most recent call last):
File "/usr/local/Cellar/[email protected]/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/Cellar/[email protected]/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.9/site-packages/pip_run/read-deps.py", line 16, in <module>
__name__ == '__main__' and run()
File "/usr/local/lib/python3.9/site-packages/pip_run/read-deps.py", line 12, in run
deps = DepsReader.load(script).read()
AttributeError: type object 'DepsReader' has no attribute 'load'
This failure appears to be due to read-deps.py
not having been updated after DepsReader.load
was split out into SourceDepsReader
in commit f943b56.
I have a scenario where if I make the following requirements.txt
file:
cryptography<=3.2.1,>2.9.2
azure-storage-blob==12.8.1
and pass it with -r
, everything is fine.
If I add this to the script i'm running, and remove the -r
flag:
__requires__ = [
'cryptography<=3.2.1,>2.9.2',
'azure-storage-blob>12',
]
I get an error because my site-packages has jwt 1.1.0
and cryptography 3.2.1
installed:
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
jwt 1.1.0 requires cryptography<=3.2.1,>2.9.2, but you have cryptography 3.4.7 which is incompatible.
What i'd like:
-r
behavior without having to make another separate fileWhat I'd love:
Originally, I created rwt
because I wanted to have a short, pithy name. I thought rwt
would be distinct and develop its own namesake and following, but it has not yet.
And since rwt
doesn't convey any meaningful purpose to users, but other projects like pipenv
and piptools
and pip-compile
have set a precedent for just prefixing the tool name with pip, I think rwt
should do the same and present as pip-run
. That will convey to users much more succinctly the purpose of the project and a little bit of what it does.
Hi @jaraco,
thanks for this great package!
I'm happily using it for bootstrapping build/test requirements in my setup.py, and replace setup_requires
and tests_requre
(I don't wanna compile cython into an .egg!)
Now, in the section Replacing setup_requires
of your README.rst (btw, why isn't Github rendeing the ReStructuredText properly?), you recommend to add setuptools itself to the __requires__
list.
I wonder, isn't this a bit redundant?
I can run python -m rwt -- setup.py ...
just fine even if I don't explicitly add setuptools to __requires__
.
Isn't setuptools a dependency of pip? And since rwt in turn needs pip to do its magic, if one uses rwt, one must also have pip and hence setuptools installed -- even only to install rwt in the first place.
I could see the use of it if, say, one requires a specific minimum version of setuptools.
But why requiring a generic one?
Sorry if my question sound stupid, and thanks again for the great work :)
In #24, this project adopted support for specifying --dependency-links
in a script to support VCS installation. Since then (or perhaps before), pip added support for VCS URLs directly in requirements. Let's remove this now deprecated functionality.
Per 3463926#r94301964, @mgorny reports that the app_paths dependency is problemmatic. Let's explore ways to fix that.
Although the PEP 518 support added by pip 10 goes a long way toward supporting the use cases of installing dependencies for the "install" and "wheel [build]" steps, it is of much less help for many of the other use-cases that setuptools setup_requires
facilitates. A few prominent ones:
But more generally, anything that invokes setup.py
for its distutils-based behaviors.
@pganssle has proposed that Setuptools should present a CLI that also supports PEP 518 for this use-case, and I'm convinced something like that is needed, as I also as hard as I try am finding it difficult to convert projects away from the tried-and-true paradigms to whole new ones without real issues arising.
But it also occurs to me that there is a project that has support for installing packages on demand and which might be a suitable bridge for setup.py
, and that's this project (RWT).
RWT already builds on pip and builds throw away environments for the duration of a command (not dissimilar to what setup-requires does). And RWT was written with supplanting many setup-requires use-cases.
What RWT doesn't have is support for PEP 518 build requirements, but since it's a thin wrapper around pip, if pip were to supply an argument to pip install
, namely --build-reqs
, RWT could build on that also, and it would remove the need for multiple packages to have build-requirement installer support.
Here's how it would work. If invoked on a directory containing a pyproject.toml (or referencing such a file), pip would install the build requirements:
(myenv) $ pip install --build-reqs .
... pip installs build requirements defined in pyproject.toml (permanently)
But in the case where a user does not wish to install the build requirements into the environment, but does wish to invoke a setup.py command with the build requirements present, they could use rwt as the tool to do so:
$ rwt --build-reqs . -- setup.py test
... rwt invokes `install --build-reqs` into a temporary directory, adds to PYTHONPATH, then invokes setup.py test.
I was originally thinking that pip
should expose a new command install-build-reqs
, but then I realized that rwt
only wraps pip install
, so a parameter to pip install
makes a lot more sense.
Alternatively, rwt
could implement this support more directly (without any modifications to pip) by adding a parameter that (a) bypasses default behavior of passing parameters to pip and (b) reads the pyproject.toml and passes the parsed deps to pip install
instead.
Advantages of this proposed approach:
pip run
as proposed in pypa/pip#3971)Disadvantages:
rwt
) to gain the functionality (barring pypa/pip#3971).rwt
design doesn't keep the build environment around the way setup-requires does, so the startup cost is paid at each invocation (a cost which is substantially mitigated in some cases by pip caches).@pganssle - if you haven't already, would you give rwt a trial run (I suggest having it installed in your global environment; it's lightweight and often immensely useful) and after let me know what you think about the proposal.
@benoit-pierre I'd be interested in your opinion also.
When sending a sigint to the command, the underlying command will exit normally, but then rwt will receive the KeyboardInterrupt as well. Where possible, rwt should probably just use execv
to replace the process.
One key feature of rwt
is that it can be readily used on a single script to resolve dependencies declared in the script and install them, using the setuptools convention of __requires__
. This convention, however, does not allow for other options to install, such as the ever-important --index-url
. There should be another, comparable mechanism for a script to declare other parameters to pip.
Q: When does a with
context never exit?
A: When the command in the context invokes execv.
It occurred to me today that using execve
to launch the child process would eliminate the cleanup logic for the temporary directory.
I started to try to think of some clever way to cause the cleanup logic to be invoked after the child process has exited:
Then I realized, I'm trying too hard and I should probably just switch rwt back to the old subprocess behavior.
I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.
python3 -sBm build -w --no-isolation
build
with --no-isolation
I'm using during all processes only locally installed modulesHere is pytest output:
+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-pip-run-9.4.0-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-pip-run-9.4.0-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra -m 'not network'
==================================================================================== test session starts ====================================================================================
platform linux -- Python 3.8.16, pytest-7.2.0, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0, configfile: pytest.ini
collected 37 items
pip_run/commands.py ... [ 8%]
pip_run/deps.py .. [ 13%]
pip_run/persist.py . [ 16%]
pip_run/read-deps.py . [ 18%]
pip_run/scripts.py .. [ 24%]
tests/test_deps.py .......... [ 51%]
tests/test_launch.py ..x.. [ 64%]
tests/test_scripts.py F..........F. [100%]
========================================================================================= FAILURES ==========================================================================================
_____________________________________________________________________________________ test_pkg_imported _____________________________________________________________________________________
tmp_path = PosixPath('/tmp/pytest-of-tkloczko/pytest-9/test_pkg_imported0')
def test_pkg_imported(tmp_path):
"""
Create a script that loads cython and ensure it runs.
"""
jaraco.path.build(
{
'script': DALS(
"""
import path
print("Successfully imported path.py")
"""
)
},
tmp_path,
)
script = tmp_path / 'script'
pip_args = ['path.py']
cmd = [sys.executable, '-m', 'pip-run'] + pip_args + ['--', str(script)]
> out = subprocess.check_output(cmd, text=True)
tests/test_scripts.py:35:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib64/python3.8/subprocess.py:415: in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
input = None, capture_output = False, timeout = None, check = True
popenargs = (['/usr/bin/python3', '-m', 'pip-run', 'path.py', '--', '/tmp/pytest-of-tkloczko/pytest-9/test_pkg_imported0/script'],), kwargs = {'stdout': -1, 'text': True}
process = <subprocess.Popen object at 0x7fa5b25f3c10>, stdout = '', stderr = None, retcode = 1
def run(*popenargs,
input=None, capture_output=False, timeout=None, check=False, **kwargs):
"""Run command with arguments and return a CompletedProcess instance.
The returned instance will have attributes args, returncode, stdout and
stderr. By default, stdout and stderr are not captured, and those attributes
will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them.
If check is True and the exit code was non-zero, it raises a
CalledProcessError. The CalledProcessError object will have the return code
in the returncode attribute, and output & stderr attributes if those streams
were captured.
If timeout is given, and the process takes too long, a TimeoutExpired
exception will be raised.
There is an optional argument "input", allowing you to
pass bytes or a string to the subprocess's stdin. If you use this argument
you may not also use the Popen constructor's "stdin" argument, as
it will be used internally.
By default, all communication is in bytes, and therefore any "input" should
be bytes, and the stdout and stderr will be bytes. If in text mode, any
"input" should be a string, and stdout and stderr will be strings decoded
according to locale encoding, or by "encoding" if set. Text mode is
triggered by setting any of text, encoding, errors or universal_newlines.
The other arguments are the same as for the Popen constructor.
"""
if input is not None:
if kwargs.get('stdin') is not None:
raise ValueError('stdin and input arguments may not both be used.')
kwargs['stdin'] = PIPE
if capture_output:
if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
raise ValueError('stdout and stderr arguments may not be used '
'with capture_output.')
kwargs['stdout'] = PIPE
kwargs['stderr'] = PIPE
with Popen(*popenargs, **kwargs) as process:
try:
stdout, stderr = process.communicate(input, timeout=timeout)
except TimeoutExpired as exc:
process.kill()
if _mswindows:
# Windows accumulates the output in a single blocking
# read() call run on child threads, with the timeout
# being done in a join() on those threads. communicate()
# _after_ kill() is required to collect that and add it
# to the exception.
exc.stdout, exc.stderr = process.communicate()
else:
# POSIX _communicate already populated the output so
# far into the TimeoutExpired exception.
process.wait()
raise
except: # Including KeyboardInterrupt, communicate handled that.
process.kill()
# We don't call process.wait() as .__exit__ does that for us.
raise
retcode = process.poll()
if check and retcode:
> raise CalledProcessError(retcode, process.args,
output=stdout, stderr=stderr)
E subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip-run', 'path.py', '--', '/tmp/pytest-of-tkloczko/pytest-9/test_pkg_imported0/script']' returned non-zero exit status 1.
/usr/lib64/python3.8/subprocess.py:516: CalledProcessError
----------------------------------------------------------------------------------- Captured stderr call ------------------------------------------------------------------------------------
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f5847c073a0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/path-py/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f5847c172e0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/path-py/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f5847c17550>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/path-py/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f5847c17700>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/path-py/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f5847c178b0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/path-py/
ERROR: Could not find a version that satisfies the requirement path.py (from versions: none)
ERROR: No matching distribution found for path.py
WARNING: There was an error checking the latest version of pip.
Traceback (most recent call last):
File "/usr/lib64/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib64/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0/pip-run.py", line 4, in <module>
__name__ == '__main__' and run()
File "/home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0/pip_run/__init__.py", line 18, in run
with deps.load(*deps.not_installed(pip_args)) as home:
File "/usr/lib64/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0/pip_run/deps.py", line 74, in load
Install.parse(args) and empty(target) and subprocess.check_call(cmd, env=env)
File "/usr/lib64/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('/usr/bin/python3', '-m', 'pip', 'install', '-t', PosixPath('/tmp/pip-run-2owvtu_x'), 'path.py')' returned non-zero exit status 1.
___________________________________________________________________________ test_pkg_loaded_from_alternate_index ____________________________________________________________________________
tmp_path = PosixPath('/tmp/pytest-of-tkloczko/pytest-9/test_pkg_loaded_from_alternate0')
def test_pkg_loaded_from_alternate_index(tmp_path):
"""
Create a script that loads cython from an alternate index
and ensure it runs.
"""
jaraco.path.build(
{
'script': DALS(
"""
__requires__ = ['path.py']
__index_url__ = 'https://devpi.net/root/pypi/+simple/'
import path
print("Successfully imported path.py")
"""
)
},
tmp_path,
)
cmd = [sys.executable, '-m', 'pip-run', '-v', '--', str(tmp_path / 'script')]
> out = subprocess.check_output(cmd, text=True)
tests/test_scripts.py:176:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib64/python3.8/subprocess.py:415: in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
input = None, capture_output = False, timeout = None, check = True
popenargs = (['/usr/bin/python3', '-m', 'pip-run', '-v', '--', '/tmp/pytest-of-tkloczko/pytest-9/test_pkg_loaded_from_alternate0/script'],), kwargs = {'stdout': -1, 'text': True}
process = <subprocess.Popen object at 0x7fa5b25373a0>, stdout = 'Looking in indexes: https://devpi.net/root/pypi/+simple/\n', stderr = None, retcode = 1
def run(*popenargs,
input=None, capture_output=False, timeout=None, check=False, **kwargs):
"""Run command with arguments and return a CompletedProcess instance.
The returned instance will have attributes args, returncode, stdout and
stderr. By default, stdout and stderr are not captured, and those attributes
will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them.
If check is True and the exit code was non-zero, it raises a
CalledProcessError. The CalledProcessError object will have the return code
in the returncode attribute, and output & stderr attributes if those streams
were captured.
If timeout is given, and the process takes too long, a TimeoutExpired
exception will be raised.
There is an optional argument "input", allowing you to
pass bytes or a string to the subprocess's stdin. If you use this argument
you may not also use the Popen constructor's "stdin" argument, as
it will be used internally.
By default, all communication is in bytes, and therefore any "input" should
be bytes, and the stdout and stderr will be bytes. If in text mode, any
"input" should be a string, and stdout and stderr will be strings decoded
according to locale encoding, or by "encoding" if set. Text mode is
triggered by setting any of text, encoding, errors or universal_newlines.
The other arguments are the same as for the Popen constructor.
"""
if input is not None:
if kwargs.get('stdin') is not None:
raise ValueError('stdin and input arguments may not both be used.')
kwargs['stdin'] = PIPE
if capture_output:
if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
raise ValueError('stdout and stderr arguments may not be used '
'with capture_output.')
kwargs['stdout'] = PIPE
kwargs['stderr'] = PIPE
with Popen(*popenargs, **kwargs) as process:
try:
stdout, stderr = process.communicate(input, timeout=timeout)
except TimeoutExpired as exc:
process.kill()
if _mswindows:
# Windows accumulates the output in a single blocking
# read() call run on child threads, with the timeout
# being done in a join() on those threads. communicate()
# _after_ kill() is required to collect that and add it
# to the exception.
exc.stdout, exc.stderr = process.communicate()
else:
# POSIX _communicate already populated the output so
# far into the TimeoutExpired exception.
process.wait()
raise
except: # Including KeyboardInterrupt, communicate handled that.
process.kill()
# We don't call process.wait() as .__exit__ does that for us.
raise
retcode = process.poll()
if check and retcode:
> raise CalledProcessError(retcode, process.args,
output=stdout, stderr=stderr)
E subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip-run', '-v', '--', '/tmp/pytest-of-tkloczko/pytest-9/test_pkg_loaded_from_alternate0/script']' returned non-zero exit status 1.
/usr/lib64/python3.8/subprocess.py:516: CalledProcessError
----------------------------------------------------------------------------------- Captured stderr call ------------------------------------------------------------------------------------
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6ac61e1280>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /root/pypi/+simple/path-py/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6ac61e1580>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /root/pypi/+simple/path-py/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6ac61e1730>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /root/pypi/+simple/path-py/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6ac61e18e0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /root/pypi/+simple/path-py/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6ac61e1a90>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /root/pypi/+simple/path-py/
ERROR: Could not find a version that satisfies the requirement path.py (from versions: none)
ERROR: No matching distribution found for path.py
WARNING: There was an error checking the latest version of pip.
Traceback (most recent call last):
File "/usr/lib64/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib64/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0/pip-run.py", line 4, in <module>
__name__ == '__main__' and run()
File "/home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0/pip_run/__init__.py", line 18, in run
with deps.load(*deps.not_installed(pip_args)) as home:
File "/usr/lib64/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0/pip_run/deps.py", line 74, in load
Install.parse(args) and empty(target) and subprocess.check_call(cmd, env=env)
File "/usr/lib64/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('/usr/bin/python3', '-m', 'pip', 'install', '-t', PosixPath('/tmp/pip-run-xz8c_iz3'), '-v', '--index-url', 'https://devpi.net/root/pypi/+simple/', 'path.py')' returned non-zero exit status 1.
================================================================================== short test summary info ==================================================================================
XFAIL tests/test_launch.py::test_with_path_overlay - cleanup can't occur with execv; #4
FAILED tests/test_scripts.py::test_pkg_imported - subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip-run', 'path.py', '--', '/tmp/pytest-of-tkloczko/pytest-9/test_pkg_imported0/script']' returned non-zero exit sta...
FAILED tests/test_scripts.py::test_pkg_loaded_from_alternate_index - subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip-run', '-v', '--', '/tmp/pytest-of-tkloczko/pytest-9/test_pkg_loaded_from_alternate0/script']' returned non-zero ...
=================================================================== 2 failed, 34 passed, 1 xfailed in 1647.11s (0:27:27) ====================================================================
Here is list of installed modules in build env
Package Version
----------------------------- -----------------
alabaster 0.7.12
appdirs 1.4.4
asn1crypto 1.5.1
attrs 22.2.0
autocommand 2.2.1
Babel 2.11.0
bcrypt 3.2.2
Brlapi 0.8.3
build 0.9.0
cffi 1.15.1
charset-normalizer 3.0.1
contourpy 1.0.6
cryptography 38.0.4
cssselect 1.1.0
cycler 0.11.0
distro 1.8.0
dnspython 2.2.1
docutils 0.19
exceptiongroup 1.0.0
extras 1.0.0
fastjsonschema 2.16.1
fixtures 4.0.0
fonttools 4.38.0
gpg 1.18.0-unknown
idna 3.4
imagesize 1.4.1
importlib-metadata 5.1.0
importlib-resources 5.10.1
inflect 5.0.0
iniconfig 1.1.1
jaraco.classes 3.2.3
jaraco.context 4.2.0
jaraco.functools 3.5.2
jaraco.packaging 9.1.1
jaraco.path 3.4.0
jaraco.text 3.11.0
jaraco.tidelift 1.5.0
jeepney 0.8.0
Jinja2 3.1.2
jsonschema 4.17.3
jupyter_core 5.1.1
keyring 23.11.0
kiwisolver 1.4.4
libcomps 0.1.19
louis 3.24.0
lxml 4.9.1
MarkupSafe 2.1.1
matplotlib 3.6.2
more-itertools 9.0.0
nbformat 5.7.0
numpy 1.23.1
olefile 0.46
packaging 21.3
path 16.6.0
pbr 5.9.0
pep517 0.13.0
Pillow 9.3.0
pip 22.3.1
pkgutil_resolve_name 1.3.10
platformdirs 2.6.0
pluggy 1.0.0
ply 3.11
pyasn1 0.4.8
pyasn1-modules 0.2.8
pycparser 2.21
Pygments 2.13.0
PyGObject 3.42.2
pyparsing 3.0.9
pyrsistent 0.19.2
pytest 7.2.0
python-dateutil 2.8.2
pytz 2022.4
PyYAML 6.0
requests 2.28.1
requests-toolbelt 0.10.1
rpm 4.17.0
rst.linker 2.3.1
scour 0.38.2
SecretStorage 3.3.2
setuptools 65.6.3
setuptools-scm 7.0.5
six 1.16.0
snowballstemmer 2.2.0
Sphinx 5.3.0
sphinxcontrib-applehelp 1.0.2.dev20221204
sphinxcontrib-devhelp 1.0.2.dev20221204
sphinxcontrib-htmlhelp 2.0.0
sphinxcontrib-jsmath 1.0.1.dev20221204
sphinxcontrib-qthelp 1.0.3.dev20221204
sphinxcontrib-serializinghtml 1.1.5
testtools 2.5.0
tomli 2.0.1
tpm2-pkcs11-tools 1.33.7
tpm2-pytss 1.1.0
traitlets 5.8.0
typing_extensions 4.4.0
urllib3 1.26.12
wheel 0.38.4
zipp 3.11.0
When invoking the Python subprocess, the exit code should be relayed back to the caller. For example:
$ python -m rwt -- -c "raise SystemExit(1)"
Loading requirements using
$ echo $?
0
Instead, 1
should be emitted.
In this tweet, I learned that pipx may in fact have substantial overlap with pip-run. I'd like to include reference to pipx and provide a fair comparison.
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/localdata/some_user/src/test_utils/venv/lib/python3.8/site-packages/pip-run.py", line 4, in <module>
__name__ == '__main__' and run()
File "/localdata/some_user/src/test_utils/venv/lib/python3.8/site-packages/pip_run/__init__.py", line 15, in run
pip_args, params = commands.parse_script_args(args)
File "/localdata/some_user/src/test_utils/venv/lib/python3.8/site-packages/pip_run/commands.py", line 58, in parse_script_args
return _separate_dash(args)
File "/localdata/some_user/src/test_utils/venv/lib/python3.8/site-packages/pip_run/commands.py", line 48, in _separate_dash
pre, post = split_at(args, '--'.__eq__, maxsplit=1)
TypeError: split_at() got an unexpected keyword argument 'maxsplit'
The maxsplit
keyword argument for split_at
was only introduced from version 8.3.0
of more-itertools
Since there is no minimum version specified in
Line 26 in c72c1dd
Then if the users environment already contains this module at a lower version, then they will encounter the error above
Suggest the requirement be updated to more_itertools>=8.3.0
?
To replicate (Python3.8):
virtualenv venv -p python3 && source ./venv/bin/activate
pip install more_itertools==8.2.0
pip install pip-run
python3 -m pip-run some_url -- -m some_module
...
TypeError: split_at() got an unexpected keyword argument 'maxsplit'
The way rwt works, it sets PYTHONPATH when invoking its child Python process, to present the new packages to it. And if that Python process invokes subprocesses, they inherit the environment by default. Sometimes, this inheritance is useful, because especially if the subprocess is the same Python (sys.executable
). Other times, it's an impediment.
Consider the case where rwt
is invoked on Python 3 and loads a number of libraries, but then the subprocess invokes a Python 2 subprocess in another environment. Now that Python 2 subprocess finds itself trying to import modules that were installed and configured for Python 3, and hilarity ensues.
I'm trying to package your module as an rpm package. So I'm using the typical build, install and test cycle used on building packages from non-root account.
Here is pytest output:
+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-pip-run-8.6.0-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-pip-run-8.6.0-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.12, pytest-6.2.5, py-1.10.0, pluggy-0.13.1
Using --randomly-seed=1944680334
rootdir: /home/tkloczko/rpmbuild/BUILD/pip-run-8.6.0, configfile: pytest.ini
plugins: forked-1.3.0, shutil-1.7.0, virtualenv-1.7.0, flake8-1.0.7, xdist-2.3.0, cov-2.12.1, mock-3.6.1, localserver-0.5.0, timeout-2.0.1, flaky-3.7.0, hypothesis-6.14.6, rerunfailures-9.1.1, checkdocs-2.7.1, anyio-3.3.1, tornasync-0.6.0.post2, randomly-3.8.0, freezegun-0.4.2, pylama-7.7.1, black-0.3.12, trio-0.7.0, twisted-1.13.3, asyncio-0.15.1
collected 32 items
pip_run/tests/test_launch.py ....x [ 16%]
pip_run/tests/test_deps.py .. [ 23%]
pip_run/deps.py . [ 26%]
pip_run/commands.py . [ 30%]
pip_run/scripts.py . [ 33%]
. . [ 36%]
pip_run/tests/test_deps.py . [ 40%]
pip_run/read-deps.py . [ 43%]
pip_run/deps.py . [ 46%]
pip_run/tests/test_deps.py . [ 50%]
pip_run/text.py . [ 53%]
pip_run/tests/test_deps.py .. [ 60%]
pip_run/tests/test_scripts.py ............ [100%]
============================================================================= warnings summary =============================================================================
pip_run/tests/test_scripts.py::TestNotebookDepsReader::test_code_and_markdown
pip_run/tests/test_scripts.py::TestNotebookDepsReader::test_code_and_markdown
pip_run/tests/test_scripts.py::TestNotebookDepsReader::test_jupyter_directives
pip_run/tests/test_scripts.py::TestNotebookDepsReader::test_jupyter_directives
pip_run/tests/test_scripts.py::TestNotebookDepsReader::test_multiple_code_blocks
pip_run/tests/test_scripts.py::TestNotebookDepsReader::test_multiple_code_blocks
pip_run/tests/test_scripts.py::TestNotebookDepsReader::test_one_code_block
/usr/lib/python3.8/site-packages/jsonschema/validators.py:190: DeprecationWarning: Passing a schema to Validator.iter_errors is deprecated and will be removed in a future release. Call validator.evolve(schema=new_schema).iter_errors(...) instead.
warnings.warn(
-- Docs: https://docs.pytest.org/en/stable/warnings.html
========================================================================= short test summary info ==========================================================================
XFAIL pip_run/tests/test_launch.py::test_with_path_overlay
cleanup can't occur with execv; #4
================================================================ 29 passed, 1 xfailed, 7 warnings in 15.99s ================================================================
Currently, when pip_run.read-deps
is passed a filename that does not exist or cannot be opened, it simply prints an empty line and exits successfully. I contend that this is the wrong behavior; such a filename is an error and should be treated as an error; an error message should be printed, and the command should exit nonzero.
When attempting to use pip-run in Setuptools tests, which turn warnings into errors, I see this error:
E pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
E
E Traceback (most recent call last):
E File "/Users/jaraco/code/public/pypa/setuptools/.tox/python/lib/python3.10/site-packages/pip_run/launch.py", line 64, in _build_env
E joined = os.pathsep.join(items)
E ResourceWarning: unclosed file <_io.TextIOWrapper name='/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-71fkg9i3/distutils-precedence.pth' mode='r' encoding='UTF-8'>
.tox/python/lib/python3.10/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning
It seems that Python warns when using open
without its context manager, such as here.
I have a number of scripts for my personal use and I found pip-run
very useful for managing my dependencies. However it still takes several seconds for pip
to install the packages, even if it already uses the download cache of pip
.
I propose adding an option (e.g. --cache
) where the directory containing the packages is not deleted after the program exits, and it can be reused as long as the dependencies list does not change. Similar tools (e.g. kotlin-main-kts
) also have caching that makes rerunning a script practically free.
There are several things that need to be considered
numpy
takes up around 70MiB). A simple solution can be keep using /tmp
and let the system manages it.I think it is best to keep the functionality simple, and in the worst case just destroy and recreate the environment. But I do expect it to be able to avoid recreation of the environment if I run the same script twice.
Currently, .pth
files are only honored on Python 3.2 and earlier due to this workaround.
Python 3.3 and later are excused because the only perceived need for .pth
files at the time of the implementation was to support namespace packages, packages which work fine in PEP 420 mode naturally on Python 3.3 and later.
However, I've since learned of other packages such as future-fstrings that install .pth
files to enable behavior on startup.
This leads me to believe rwt
should no longer exclude later Python versions from the .pth
file support... although this increases the impact of the issue that all rwt
invocations will now mask any existing sitecustomize.py
.
Something like:
__requires__ = 'dependency'
is supported.
In fact, looking at the pkg_resources
implementation for parse_requirements
and yield_lines
, something like:
__requires__ = '''
# Comment...
Babel # for translations
pyqt-distutils
PyQt5\
>=5.8.2
setuptools-scm
setuptools>=30.3.0
'''
should work too.
Feature request: Give python -m pip_run.read-deps
a --newline
(or whatever you want to call it) option that would cause the individual requirements to be output on separate lines. This would make it easy to create requirements.txt
files for __requires__
-using scripts, especially when dealing with requirements with markers like importlib-metadata>=3.6; python_version < "3.10"
, which are currently output with spaces embedded, making it difficult to separate them from adjacent requirements.
When invoking rwt on debian, the command fails thus:
$ python -m rwt requests
/usr/bin/python: No module named _vendor; 'rwt' is a package and cannot be directly executed
The issue is almost certainly the from pip._vendor import pkg_resources
in deps.py.
I tried py -m rwt --help
to get basic usage information, and I appear to get pip's help. rwt should support a standard help invocation.
Since the fix for #28, attempting to install a package from a local checkout results in nothing being installed:
$ rwt ~/p/twine
Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 26 2018, 23:26:24)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import twine
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'twine'
Somewhere between versions 2.15.1 and 3.0, rwt stopped supporting (specifically, it appears to flat-out ignore) package specifiers of the form git+ssh://[email protected]/repo.git
in __requires__
lists, though it continues to support them in requirements.txt
files. I see no mention of this change in CHANGES.rst
; was it intentional? If not, could we have the old behavior back?
Distinct from #1, where the namespace package isn't recognized at all, there's another issue that affects all versions of Python.
$ python -m rwt backports.functools_lru_cache -- -c "import backports.functools_lru_cache"
Loading requirements using backports.functools_lru_cache
$ python -c "import backports"
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named 'backports'
$ python -m pip install backports.unittest_mock
Collecting backports.unittest_mock
Downloading backports.unittest_mock-1.1.1-py2.py3-none-any.whl
Installing collected packages: backports.unittest-mock
Successfully installed backports.unittest-mock-1.1.1
$ python -c "import backports"
$ python -m rwt backports.functools_lru_cache -- -c "import backports.functools_lru_cache"
Loading requirements using backports.functools_lru_cache
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named 'backports.functools_lru_cache'
The problem is that the -nspkg.pth handler is interfering with the PEP-420 loader. Remove it and both packages import nicely.
$ rm /python/lib/python3.5/site-packages/backports.unittest_mock-1.1.1-py3.5-nspkg.pth
$ python -m rwt backports.functools_lru_cache -- -c "import backports.functools_lru_cache; import backports.unittest_mock"
Loading requirements using backports.functools_lru_cache
I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.
python3 -sBm build -w --no-isolation
build
with --no-isolation
I'm using during all processes only locally installed modules9.0.0 was OK in exactly the same build env.
Here is pytest output:
+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-pip-run-9.1.0-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-pip-run-9.1.0-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.16, pytest-7.2.0, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/pip-run-9.1.0, configfile: pytest.ini
collected 31 items
pip_run/commands.py .F. [ 9%]
pip_run/deps.py .. [ 16%]
pip_run/read-deps.py . [ 19%]
pip_run/scripts.py . [ 22%]
pip_run/text.py . [ 25%]
pip_run/tests/test_deps.py ...... [ 45%]
pip_run/tests/test_launch.py ..x.. [ 61%]
pip_run/tests/test_scripts.py ............ [100%]
================================================================================= FAILURES =================================================================================
_______________________________________________________________ [doctest] pip_run.commands._separate_script ________________________________________________________________
011
012 Inject a double-dash before the first arg that appears to be an
013 extant Python script.
014
015 >>> _separate_script(['foo', 'bar'])
016 [['foo', 'bar'], []]
017 >>> _separate_script(['foo', 'pip-run.py', 'bar'])
Expected:
[['foo'], ['pip-run.py', 'bar']]
Got:
[['foo', 'pip-run.py', 'bar'], []]
/home/tkloczko/rpmbuild/BUILD/pip-run-9.1.0/pip_run/commands.py:17: DocTestFailure
============================================================================= warnings summary =============================================================================
pip_run/tests/test_scripts.py::TestNotebookDepsReader::test_one_code_block
/home/tkloczko/rpmbuild/BUILD/pip-run-9.1.0/pip_run/scripts.py:100: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/pytest-of-tkloczko/pytest-690/test_one_code_block0/test_one_code_block.ipynb' mode='r' encoding='UTF-8'>
doc = json.load(open(script_path))
Enable tracemalloc to get traceback where the object was allocated.
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
pip_run/tests/test_scripts.py::TestNotebookDepsReader::test_multiple_code_blocks
/home/tkloczko/rpmbuild/BUILD/pip-run-9.1.0/pip_run/scripts.py:100: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/pytest-of-tkloczko/pytest-690/test_multiple_code_blocks0/test_multiple_code_blocks.ipynb' mode='r' encoding='UTF-8'>
doc = json.load(open(script_path))
Enable tracemalloc to get traceback where the object was allocated.
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
pip_run/tests/test_scripts.py::TestNotebookDepsReader::test_code_and_markdown
/home/tkloczko/rpmbuild/BUILD/pip-run-9.1.0/pip_run/scripts.py:100: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/pytest-of-tkloczko/pytest-690/test_code_and_markdown0/test_code_and_markdown.ipynb' mode='r' encoding='UTF-8'>
doc = json.load(open(script_path))
Enable tracemalloc to get traceback where the object was allocated.
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
pip_run/tests/test_scripts.py::TestNotebookDepsReader::test_jupyter_directives
/home/tkloczko/rpmbuild/BUILD/pip-run-9.1.0/pip_run/scripts.py:100: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/pytest-of-tkloczko/pytest-690/test_jupyter_directives0/test_jupyter_directives.ipynb' mode='r' encoding='UTF-8'>
doc = json.load(open(script_path))
Enable tracemalloc to get traceback where the object was allocated.
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================================================================= short test summary info ==========================================================================
XFAIL pip_run/tests/test_launch.py::test_with_path_overlay - cleanup can't occur with execv; #4
FAILED pip_run/commands.py::pip_run.commands._separate_script
=========================================================== 1 failed, 29 passed, 1 xfailed, 4 warnings in 7.16s ============================================================
Here is list of installed modules in build env
Package Version
----------------------------- -----------------
alabaster 0.7.12
appdirs 1.4.4
asn1crypto 1.5.1
attrs 22.1.0
autocommand 2.2.1
Babel 2.11.0
bcrypt 3.2.2
Brlapi 0.8.3
build 0.9.0
cffi 1.15.1
charset-normalizer 3.0.1
contourpy 1.0.6
cryptography 38.0.4
cssselect 1.1.0
cycler 0.11.0
distro 1.8.0
dnspython 2.2.1
docutils 0.19
exceptiongroup 1.0.0
extras 1.0.0
fastjsonschema 2.16.1
fixtures 4.0.0
fonttools 4.38.0
gpg 1.18.0-unknown
idna 3.4
imagesize 1.4.1
importlib-metadata 5.1.0
importlib-resources 5.10.1
iniconfig 1.1.1
jaraco.classes 3.2.3
jaraco.packaging 9.1.1
jaraco.tidelift 1.5.0
jeepney 0.8.0
Jinja2 3.1.2
jsonschema 4.17.3
jupyter_core 5.1.0
keyring 23.11.0
kiwisolver 1.4.4
libcomps 0.1.19
louis 3.24.0
lxml 4.9.1
MarkupSafe 2.1.1
matplotlib 3.6.2
more-itertools 9.0.0
nbformat 5.7.0
numpy 1.23.1
olefile 0.46
packaging 21.3
path 16.6.0
pbr 5.9.0
pep517 0.13.0
Pillow 9.3.0
pip 22.3.1
pkgutil_resolve_name 1.3.10
platformdirs 2.6.0
pluggy 1.0.0
ply 3.11
pyasn1 0.4.8
pyasn1-modules 0.2.8
pycparser 2.21
Pygments 2.13.0
PyGObject 3.42.2
pyparsing 3.0.9
pyrsistent 0.19.2
pytest 7.2.0
python-dateutil 2.8.2
pytz 2022.4
PyYAML 6.0
requests 2.28.1
requests-toolbelt 0.10.1
rpm 4.17.0
rst.linker 2.3.1
scour 0.38.2
SecretStorage 3.3.2
setuptools 65.6.3
setuptools-scm 7.0.5
six 1.16.0
snowballstemmer 2.2.0
Sphinx 5.3.0
sphinxcontrib-applehelp 1.0.2.dev20221204
sphinxcontrib-devhelp 1.0.2.dev20221204
sphinxcontrib-htmlhelp 2.0.0
sphinxcontrib-jsmath 1.0.1.dev20221204
sphinxcontrib-qthelp 1.0.3.dev20221204
sphinxcontrib-serializinghtml 1.1.5
testtools 2.5.0
tomli 2.0.1
tpm2-pkcs11-tools 1.33.7
tpm2-pytss 1.1.0
traitlets 5.4.0
typing_extensions 4.4.0
urllib3 1.26.12
wheel 0.38.4
zipp 3.11.0
When piping input to a rwt-launched process, the stdin is not piped so any content piped is lost at the rwt invocation.
In pypa/pip#3971 (comment), @pfmoore has to say:
I find having to include the
--
to separate the command parameters clumsy, and I'm not in love with details like the default being to print all of pip's output. Which all adds up to a general sense of "I never think to use it, because it's sort of alright, but irritating". Sorry, I know that's annoyingly vague, and I do think that the UI is the hardest part of this to get right. What I want is to be able to typepip-run my_script.py arg1 arg2 arg3
, and have it run my script with all dependencies available. The other features ofpip-run
are secondary (to me).
In this issue, I'll focus on the double-dash behavior.
The reason for the --
is to clearly separate the parameters to the script from parameters to pip/pip-run. Since there are two concerns to deal with here, there needs to be clear separation. For me, 95% of the usage is on the install parameters side, with the right hand side accepting any argument to Python (sometimes no args (for a repl), sometimes -c
, sometimes -m
, and sometimes a script). This generalization means the tool has a wide variety of applications.
I'd like to improve the reported scenario, but I struggle a bit with how to do that without drastically degrading the current experience.
One possible improvement could be to allow the --
to be optional in the specific case where a script is the first parameter. That is, if the first parameter is an extant file, that's not a valid parameter to pip install
, so pip-run
could infer a --
before it. If someone wanted to pass arguments to the interpreter or to pip, they would need to bring back the --
, but for convenience, a simple script invocation could omit it. WDYT?
In pypa/pip#3971, key members of the PyPA have leveled critiques of the embedded requirements feature, where a user can supply requirements and other common installer directives to signal to the tool how (best) to run the script. This feature emerged from the use-case where a script author would like to distribute a single file and have that file be runnable by any number of end users with minimal guidance, allowing the file to be published to a gist or alongside other scripts in a directory without needing additional context for executability.
--extra-index-url
or --quiet
.I gave this a try, and my immediate thought was to run it as "rwt". I know the docs say to use "python -m rwt", but maybe add a console entry point as a convenience?
In Pip 9.x and earlier, pip install
with no requirements would issue a warning. In pip 10, that's an error. Since rwt filters out packages already satisfied, it may end up with an empty set, triggering this warning. If no requirements are indicated, it should avoid the pip install invocation and the emergent error.
Due to pypa/pip#4106, pip install --target
doesn't work on Homebrew Python (or other Pythons that set prefix). As a result, rwt is nonviable in those environments.
In this comment, silverwind provides a workaround. Perhaps rwt can employ that workaround.
Install a package like "newrelic", which specifies this in its setup.py:
extra_path = ( "newrelic", "newrelic-%s" % package_version ),
When pip invokes setup.py install --single-version-externally-managed
on the package, it generates this in the lib:
$ ls foo/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages
newrelic-2.68.0.50 newrelic.pth
That is, the package is installed into newrelic-2.68.0.50 with a newrelic.pth file pointing at that dir. As a result, pointing PYTHONPATH at that install directory, which won't honor .pth files, won't find newrelic as a dependency.
Reading up, it appears as if extra_path is an undocumented feature.
Hey @jaraco. I am reading about the project and for some reason I want to believe that rwt
is able to figure out and fetch dependencies automatically, but so far I haven't found proof of that - we still need to create a custom "import" section just for rwt
. In this respect rwt
is still manual dependency management like http://docs.groovy-lang.org/latest/html/documentation/grape.html
In #14, a workaround was added to support homebrew and other environments that override the prefix
. In pypa/pip#6008, a fix was merged to pip to support --target
in those environments and released with pip 19.2. That means this project can drop that workaround as long as it requires pip 19.2 or later.
The code currently has an implicit dependency on pkg_resources
, which is satisfied by its dependency on pip (which vendors that pkg_resources). It would be nice to remove this dependency. By addressing #39, the only remaining usage of pkg_resources is on determining which package versions are already satisfied. I considered using importlib_metadata
to do that, but ran into python-devs/importlib_metadata#82.
Consider:
~ $ rwt -q inflect
ERROR: You must give at least one requirement to install (see "pip help install")
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/bin/rwt", line 11, in <module>
sys.exit(run())
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/rwt/__init__.py", line 15, in run
with deps.load(*deps.not_installed(pip_args)) as home:
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 112, in __enter__
return next(self.gen)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/rwt/deps.py", line 49, in load
args and subprocess.check_call(cmd)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 328, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('/Library/Frameworks/Python.framework/Versions/3.7/bin/python3', '-m', 'pip', 'install', '-t', '/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/rwt-dwyx_rwx', '-q')' returned non-zero exit status 1.
In #25, we addressed this issue for simple declarations to pip, but it seems not to be working when other parameters like -q
are present.
Due to the way pip installs namespace packages, relying on .pth files to satisfy namespace packages, and because .pth files are only honored in site-packages, rwt cannot load packages that are namespace packages on older Pythons like 2.7.
$ python3.5 -m rwt backports.functools_lru_cache -- -c "import backports"
Loading requirements using backports.functools_lru_cache
$ python2.7 -m rwt backports.functools_lru_cache -- -c "import backports"
Loading requirements using backports.functools_lru_cache
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named backports
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.