sio / makefile.venv Goto Github PK
View Code? Open in Web Editor NEWSeamlessly manage Python virtual environment with a Makefile
Home Page: https://pypi.org/project/Makefile.venv/
License: Apache License 2.0
Seamlessly manage Python virtual environment with a Makefile
Home Page: https://pypi.org/project/Makefile.venv/
License: Apache License 2.0
venv is a .PHONY target and rules that depend on it will be executed every time make is run. This behavior is sensible as default because most Python projects use Makefiles for running development chores, not for artifact building. In cases where that is not desirable use order-only prerequisite syntax:
For the majority of my use, venv being phony is fine, as I can set it as an order-only dependency.
But I have a make rule that invokes pyinstaller, and so I want my output artefacts to be rebuilt if venv changes. What's the best way to do this? For now I've just gone for $(VENV)/$(MARKER)
, but is that the best solution? (Especially as it means I have to include Makefile.venv
near the top)
Hi, I stumbled upon a problem: Recently Windows user tend to use the Python Launcher for Windows to execute python programs. So they use, e.g., py -3 script.py
instead of python3 script.py
to launch a script with Python 3.
Unfortunately that also means that the python executable is not be on the path with its *nix/Linux name.
This leads to a problem due to PY
using the *nix/Linux name
Lines 44 to 47 in bbb6795
py
, so this will failLines 90 to 93 in bbb6795
It is very annoying that Windows continuously creates problems for cross-platform tools (like Makefile.venv). I don't have a good idea how to solve that. Maybe detect if py.exe
can be found on PATH
before the OS detection and if so replace 'python' with 'py -' in PY
? What do you think?
I built this snippet that seems to work:
# Detect Windows py launcher and use it to find the given python version
ifeq ($(OS),Windows_NT)
ifneq (, $(shell where py 2>NUL))
PY := $(subst python,py -,$(PY))
endif
endif
Some more infos:
py -3.7 -m venv .venv
results in a proper .venv/
with python.exe
and pip.exe
in .venv/Scripts/
py -3.7 -c "import __future__, sys; print(sys.platform)"
prints win32
I had to replace the block where OS is detected in order to support cygwin.
I believed that the following doesn't break anything, but you may want to double check before doing the change:
VENV=$(VENVDIR)/bin
EXE=
ifdef OS # Windows support
ifneq ($(shell echo $$OSTYPE),cygwin) #Cygwin support
VENV=$(VENVDIR)/Scripts
EXE=.exe
endif #OSTYPE
endif #OS
Scheduled test runs have started failing today on windows-latest
:
This run was using:
Python 3.9.10 (tags/v3.9.10:f2f3f53, Jan 17 2022, 15:14:21) [MSC v.1929 64 bit (AMD64)]
Windows-10-10.0.20348-SP0
GNU Make 4.2.1
This run was using:
Python 3.7.9 (tags/v3.7.9:13c94747c7, Aug 17 2020, 18:58:18) [MSC v.1900 64 bit (AMD64)]
Windows-10-10.0.17763-SP0
GNU Make 4.2.1
The obvious change in CI runner is Python upgrade from 3.7.9 to 3.9.10. The failure is not inherent to that Python version: macos and cygwin were already using it successfully in the same run.
Rerun eliminated test failures related to command timeouts, but this one has remained:
FAIL: test_setup_cfg (tests.test_dependencies.TestDependencies)
Check that setup.cfg is being processed correctly
----------------------------------------------------------------------
Traceback (most recent call last):
File "D:\a\Makefile.venv\Makefile.venv\tests\test_dependencies.py", line 33, in test_setup_cfg
make = self.make('hello', makefile=makefile)
File "D:\a\Makefile.venv\Makefile.venv\tests\common.py", line 62, in make
self.check_returncode(process, returncode)
File "D:\a\Makefile.venv\Makefile.venv\tests\common.py", line 67, in check_returncode
self.assertEqual(
AssertionError: 2 != 0 : make exited with code 2 (expected 0)
stdout:
mingw32-make[1]: Entering directory 'C:/Users/RUNNER~1/AppData/Local/Temp/Makefile.venv_test_7o0hpfn3'
./.venv/Scripts/pip install -e ./
Obtaining file:///C:/Users/RUNNER~1/AppData/Local/Temp/Makefile.venv_test_7o0hpfn3
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
Requirement already satisfied: pyfiglet in c:\users\runneradmin\appdata\local\temp\makefile.venv_test_7o0hpfn3\.venv\lib\site-packages (from hello==0.0.0) (0.8.post1)
Installing collected packages: hello
Attempting uninstall: hello
Found existing installation: hello 0.0.0
mingw32-make[1]: Leaving directory 'C:/Users/RUNNER~1/AppData/Local/Temp/Makefile.venv_test_7o0hpfn3'
stderr:
ERROR: Exception:
Traceback (most recent call last):
File "C:\Users\RUNNER~1\AppData\Local\Temp\Makefile.venv_test_7o0hpfn3\.venv\lib\site-packages\pip\_internal\cli\base_command.py", line 167, in exc_logging_wrapper
status = run_func(*args)
File "C:\Users\RUNNER~1\AppData\Local\Temp\Makefile.venv_test_7o0hpfn3\.venv\lib\site-packages\pip\_internal\cli\req_command.py", line 205, in wrapper
return func(self, options, args)
File "C:\Users\RUNNER~1\AppData\Local\Temp\Makefile.venv_test_7o0hpfn3\.venv\lib\site-packages\pip\_internal\commands\install.py", line 405, in run
installed = install_given_reqs(
File "C:\Users\RUNNER~1\AppData\Local\Temp\Makefile.venv_test_7o0hpfn3\.venv\lib\site-packages\pip\_internal\req\__init__.py", line 68, in install_given_reqs
uninstalled_pathset = requirement.uninstall(auto_confirm=True)
File "C:\Users\RUNNER~1\AppData\Local\Temp\Makefile.venv_test_7o0hpfn3\.venv\lib\site-packages\pip\_internal\req\req_install.py", line 637, in uninstall
uninstalled_pathset = UninstallPathSet.from_dist(dist)
File "C:\Users\RUNNER~1\AppData\Local\Temp\Makefile.venv_test_7o0hpfn3\.venv\lib\site-packages\pip\_internal\req\req_uninstall.py", line 530, in from_dist
assert link_pointer == dist_location, (
AssertionError: Egg-link c:\users\runner~1\appdata\local\temp\makefile.venv_test_7o0hpfn3 does not match installed location of hello (at c:\users\runneradmin\appdata\local\temp\makefile.venv_test_7o0hpfn3)
mingw32-make[1]: *** [D:\a\Makefile.venv\Makefile.venv\Makefile.venv:223: .venv/Scripts/.initialized-with-Makefile.venv] Error 2
The last part indicates that the issue is caused by pip failing to correctly compare shortened Windo~1 path to a full version:
AssertionError:
Egg-link c:\users\runner~1\appdata\local\temp\makefile.venv_test_7o0hpfn3
does not match installed location of hello
(at c:\users\runneradmin\appdata\local\temp\makefile.venv_test_7o0hpfn3)
I'll wait this out a little in hopes that upstream pip will fix the issue in the next update. Will check pip bug reports now.
I want to override the virtual environment dir to include the Python version on the path.
Here's the relevant part of my Makefile:
PYTHON_VERSION := 3.7
PY := python$(PYTHON_VERSION)
VENVDIR := .venv/$(PYTHON_VERSION)
include release/Makefile.venv
And here is the output for a simple command
$ make show-venv
release/Makefile.venv:134: warning: overriding recipe for target '.venv/3.7'
release/Makefile.venv:129: warning: ignoring old recipe for target '.venv/3.7'
release/Makefile.venv:174: warning: overriding recipe for target '.venv/3.7'
release/Makefile.venv:134: warning: ignoring old recipe for target '.venv/3.7'
release/Makefile.venv:173: *** mixed implicit and normal rules: deprecated syntax
make: Circular .venv/3.7 <- .venv/3.7 dependency dropped.
make: Circular /bin/.initialized-with-Makefile.venv <- .venv/3.7 dependency dropped.
.venv/3.7 /bin/pip install -r requirements.txt
make: .venv/3.7: No such file or directory
make: *** [release/Makefile.venv:134: /bin/.initialized-with-Makefile.venv] Error 127
Hardcoding the VENVDIR
without variables doesn't help, and neither does turning it into a regular variable (VENVDIR = .venv/3.7
).
Maybe I'm doing something wrong?
I'm using version "v2020.05.05"
currently Makefile.venv tries to install the latest pip version at venv creation time
Lines 136 to 138 in abfec93
But we found that having an arbitrary version can cause problems. The use case is when packaging wheel files so that we can have a local, "wheel-only" install (i.e. no indexes or network connection at install time).
pip wheel
is used, and the selected whl files are determined by the pip version(for example, see the table at https://github.com/pypa/manylinux)
So it may be useful to have an option to pin the pip version.
GitHub Actions provide Windows environment with half of Unix installed on top. This is not exactly representative of an average Windows PC.
We should run CI tests in a more bare environment: just Python from python.org, some build of GNU Make for orchestration (choco?) and cmd.exe for shell. There should be no touch
command, no /dev/null
to write to, no sh.exe
to fall back to... Code path for such environment will be significantly different to what we test now, and some new issues may be discovered.
One way to achieve this would be to use Docker Windows containers.
As with many things Windows, running Docker is annoying:
Currently I have no patience for this obnoxious stuff. Leaving it here in case someone else is interested.
It would be wonderful if there were a way to specify that optional dependencies are to be installed; e.g., in my pyproject.toml
, I have:
[project.optional-dependencies]
dev = ["mypy", "black", "pylint", "pytest", "pytest-md", "validate-pyproject"]
This would allow projects to avoid having a separate requirements.txt
file if they choose not to.
hi, thanks for the nice timesaving package - would you happen to know how I can integrate usage of pyproject.toml over requirements.txt?
Hi there, thanks for this cool tool :)
Is there any way to pass additional arguments to the pip install
that installs the requirements files? Specifically, my workflow involves dependencies that are in a private repository so I need to pass --extra-index-url
.
Looking at the source it seems it is not possible at moment. Perhaps something like this could do the job?
REQUIREMENTS_TXT?=requirements.txt # Multiple paths are supported (space separated)
PIP_INSTALL_ARGS?=
...
ifneq ($(_REQUIREMENTS),)
$(VENV)/pip install $(foreach path,$(_REQUIREMENTS),-r $(path)) $(PIP_INSTALL_ARGS)
endif
...
If the above is sensible I can throw a PR :)
Hi, when Makefile.venv
creates a venv
, it does this:
Lines 144 to 146 in 9eae972
I have a Makefile
to automate the build process. It needs to do some steps before the build, otherwise the build will fail. I don't really understand why Makefile.venv
tries to install the local project. In my eyes, Makefile.venv
manages the venv
using requirements.txt
and the other Makefile
manages the build of the project. Am I misinterpreting something?
I'm using Big Sur 11.2.
Somehow, sha256sum
is not pre-installed on my machine. So I have to change from:
echo "5afbcf51a82f629cd65ff23185acde90ebe4dec889ef80bbdc12562fbd0b2611 *Makefile.fetched" \
| sha256sum --check - \
&& mv Makefile.fetched Makefile.venv
to:
echo "5afbcf51a82f629cd65ff23185acde90ebe4dec889ef80bbdc12562fbd0b2611 *Makefile.fetched" \
| shasum -a 256 --check - \
&& mv Makefile.fetched Makefile.venv
Wonder if it should be updated...
If you run make
with -R
, there is no value for $(RM)
set, which causes clean-venv
to fail. This is probably worth fixing with a simple ?=
rule so that this works without relying on default variable values.
After using this in several projects (thanks for sharing this!), I found myself needing project-specific shell scripts for some tasks.
I wanted to re-use the venv created by this tool, since it has all the necessary tools already installed, but this is not easy to do, since I'm writing a shell script, not Python code. After some trial and error, I found this to work very well:
On the Makefile
, use:
export VENV
On the scripts themselves, I use this snippet on the top:
#!/bin/bash
if [ -n "$VENV" ]; then
echo "# Augment PATH with '$VENV'" >&2
PATH="$(realpath "$VENV"):$PATH"
export PATH
fi
That last export
re-exports the PATH to child processes, so that they can also use the venv tools. These scripts also work inside the shell from make bash
, with duplicated PATH.
Another (worse) alternative is to adjust all calls, using something like this:
"${VENV+$VENV/}python" --version
That way you can define target in the "usual" way in the Makefile
, and all the venv tools are available on the PATH:
output.txt: input.txt | $(VENV)
location/script <$< >$@
This requires changes on the scripts themselves, but I think this is the cleanest way.
Just putting this out here in case anyone runs into the same problem.
First, let me say I love this project. I started down this road myself a few hours before finding it. Super helpful, saved me a lot of time, and showed me things I hadn't even thought of yet.
I'm using GitHub actions which are dependent on environment variables that should not be stored in git. A local .env file which ii sourced by targets while testing supports this workflow.
Enabling the shell targets to source a .env file would remove a step if doing more interactive manual testing.
It seems normal for projects to make custom rules for generating *.txt from *.in, and such a rule will need to use venv for pip-compile.
However when we tried it, it creates a circular dependency since Makefile.venv has a venv dependency on the REQUIREMENTS_TXT
files.
Can Makefile.venv support this use case?
Perhaps Makefile.venv is doing too much implicitly-- could it do less and do it explicitly?
cc: @alegonz
Cygwin behaves quite differently from both Windows and Linux. Running CI tests in Cygwin seems reasonable.
Related: #2
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.