Trying to install the local project isn't necessarily desired about makefile.venv HOT 7 CLOSED

While this may be a failure on documentation part, the behavior is most definitely intended.

There are many ways to define Python project dependencies and not one of them is official or blessed. That's one of the primary reasons I have created Makefile.venv. There are recent developments in that area but they are still at risk of going xkcd 927 route.

Both setup.py and requirements.txt are valid ways to define Python project dependencies and they were both highly recommended at some stage and are both well adopted by the Python community. There is no reason to treat them differently. The fact that you use them at the same time and assign different semantics to them is not universal. It's a design decision of your project and it's not applicable to a generic case.

Makefile.venv aims to be an one-stop solution for Python virtual environment management, regardless of the format used to define that venv. That's why we have docs for pip-compile and stubs for pipenv/poetry. Setup.py and requirements.txt are supported out of the box because they have become de-facto standards, but if anything else will take their place - Makefile.venv will support that too. It was never just a "requirements.txt manager".

In your case Makefile.venv sees both requirements.txt and setup.py and it knows that venv is not complete until both those files are processed. If processing setup.py requires some prior steps to be executed, you should provide recipes for them and define appropriate dependencies between targets (Makefile.venv is not a standalone Makefile, it's meant to be included from one, i.e. library, not application). Here is a rough example:

special-step1:
    install some complex toolchain

special-step2: special-step1
    configure the complex toolchain

$(VENV): | special-step1 special-step2

In my eyes, Makefile.venv manages the venv using requirements.txt and the other Makefile manages the build of the project. Am I misinterpreting something?

You may be. Makefile.venv provides a way for your Makefile to manage the venv with little to no headaches. It's not a standalone tool.

Also, the "build" term is kinda vague when we're speaking of Python. Virtual environment is not "built", it's created by installing packages. The fact that some packages require compilation steps to be installed is abstracted away by multiple layers from the concept of venv.

For what purpose do you intend to use Makefile.venv? How do you decide which dependencies to list in requirements.txt and which in setup.py? Why is partially configured venv (with packages only from requirements.txt) useful to you?

from makefile.venv.

I've added the changes to feature branch, everything seems to be ok.

Some documentation needs to be updated before merging this to master, but the code itself should stay the same.

What do you think?

from makefile.venv.

I've merged the changes into master and pushed a new release: v2021.12.01

Thank you for reporting this!

from makefile.venv.

Thanks for your detailed explanation, it already made things much clearer for me. In almost all points I fully agree with your rational and support the decisions.

The fact that you use them [requirements.txt and setup.py] at the same time and assign different semantics to them is not universal. It's a design decision of your project and it's not applicable to a generic case.

You are right, we assumed a general consensus on the usage of requirements.txt and setup.py.

I know it is a lot to ask, but do you think it would be possible to support our interpretation of requirements.txt and setup.py as well? Basically we would need a flag to ignore the setup.py. I think right now it is possible to ignore the requirements.txt (by setting REQUIREMENTS_TXT to a non existing file), but setup.py is always used.

Maybe a few points about the reason why we chose our setup and why we think it is a reasonable choice:

• From the C world I am used to a Makefile being used something like this: ./configure && make test && make build && make install. Makefile.venv is for me the part that does the ./configure, i.e., create an environment that can be used for the build process with make.
• If I understand correctly you do the pip install -e . to install the dependencies from install_requires in the setup.py. However, this also installs the project to the venv (which I assume is just a byproduct?).
• In my understanding, the install_requires in the setup.py is for runtime dependencies of the project, saying, what other packages do I need to have installed to run this package. However, the Makefile is used for the build process not for running the project.
• Our interpretation was, that requirements.txt specifies the build environment, so if something is required to execute any target (e.g., test, build, install, ...), it should be in requirements.txt. If it is required for running the project, it should be in the install_requires in the setup.py.

In your case Makefile.venv sees both requirements.txt and setup.py and it knows that venv is not complete until both those files are processed.

I would disagree here, you can build the project with setuptools without installing what is in install_requires. But yes, if you use the Makefile to run the project, you would need the packages from install_requires.

For what purpose do you intend to use Makefile.venv? How do you decide which dependencies to list in requirements.txt and which in setup.py? Why is partially configured venv (with packages only from requirements.txt) useful to you?

I think my response above covers the parts on my intent, but let me elaborate on the actual use case to give an example. We use setuptools' setup.py to configure the packet bundling process, and requirements.txt to manage the dependencies for development, test, and build. For example, the build will also generate the documentation with sphinx, and we have a test target that does some linting and type checking. So requirements.txt contains something like flake8, mypy, sphinx, ... (of course those are no runtime dependencies of the project). The idea is, you should be able to clone the repo and do, e.g., a make test. It will create a venv for you and install all the requirements (Makefile.venv will do that) and start the tests (Makefile will do that). The make build is working the same, but it will generate some files that are included in setup.py, like the version string. Our build target also depends on the venv but with pip install -e ., Makefile.venv will try to build the setup.py before those files are created. That fails.

I think your proposed solution will work. However, that pushes part of the build steps before anything else (even the tests), which is a bit hacky.

from makefile.venv.

I see. Basically you have two environments: development environment and runtime environment. Sometimes they happen to coexist within one Python virtual env, sometimes only one of them is present/required.

I've encountered such scenario before but in my case there were no downsides in adding runtime dependencies to development environment, so I haven't put more thought in this. I agree that your concerns are valid and that Makefile.venv currently treats requirements.txt and setup.py unequally.

I think right now it is possible to ignore the requirements.txt (by setting REQUIREMENTS_TXT to a non existing file), but setup.py is always used.

Thanks! You've pointed out another documentation miss! 😃 Setting REQUIREMENTS_TXT to a non existent path will result in an error (No rule to make target 'nonexistent', needed by '.venv/bin/.initialized-with-Makefile.venv') - that's the consequence of adding pip-compile compatibility I've mentioned earlier. But users can still ignore requirements.txt by providing an empty value for that variable.

I think we can apply similar approach to processing setup.py. Coincidentally there is a years-old note in my phone:

Move setup.py path to variable - do we gain anything by this?

Turns out we do. I'll experiment a little and will come back later.

from makefile.venv.

I'll experiment a little and will come back later.

That sounds great! 🥇

Setting REQUIREMENTS_TXT to a non existent path will result in an error (No rule to make target 'nonexistent', needed by '.venv/bin/.initialized-with-Makefile.venv')

To be honest I was just speculating on this part based on the line below and didn't try it out. Sorry about that. 😇 I'm glad the idea is still valid even though I was wrong.

from makefile.venv.

That was fast! 🚀 And it looks splendid. 👍 This should allow us to ignore the setup.py and specify the venv just via the requirements.txt. Thank you very much!

from makefile.venv.