Isolated build environment has access to build's environment when run from a Python environment about build HOT 14 CLOSED

Can you not use the Python -S flag to prevent the import of the site module, which will prevent the parsing of pyvenv.cfg? You can then manipulate PYTHONPATH normally, by breaking out of the virtual environment. Although not importing site might have other implications that I'm not aware of.

from build.

You mean PEP 405 was a breaking change? When PEPs propose a change Python semantics, they're never implemented in minor versions. Looks like PEP 405 was introduced in Python 3.3 (Python does not use semver — backwards-incompatible changes are allowed in point releases like 3.2 → 3.3).

Yes, but AFAIK you don't change behaviors, you deprecate stuff and remove them, but I might be wrong. Changing how something behaves is very problematic.

In this case, I see absolutely no reason for pyvenv.cfg to be the first step of the search.

This is a somewhat vague description of the problem. I don't really understand the isolation / environment code well enough to understand why this would be a problem. From what I can tell, PYTHONHOME refers to the location of the standard library, but the problem we're having is not from the standard library, it's from site-packages. I'll note that it's entirely possible to prevent access to these by removing the relevant directories from sys.path (though I'm not sure if it's possible to do that with environment variables).

Okay, this is the issue:

(on /usr/bin/python, no pyvenv.cfg is found)

$ PYTHONHOME=/tmp/build-env-kit8jz7e PYTHONPATH=/usr/lib/python38.zip:/usr/lib/python3.8:/usr/lib/python3.8/lib-dynload:/usr/lib/python3.8/site-packages:/tmp/build-env-kit8jz7e/lib/python3.8/site-packages:/tmp/build-env-kit8jz7e/lib/python3.8/site-packages /usr/bin/python -m sysconfig | head -n 15
Platform: "linux-x86_64"
Python version: "3.8"
Current installation scheme: "posix_prefix"

Paths:
	data = "/tmp/build-env-kit8jz7e"
	include = "/tmp/build-env-kit8jz7e/include/python3.8"
	platinclude = "/tmp/build-env-kit8jz7e/include/python3.8"
	platlib = "/tmp/build-env-kit8jz7e/lib/python3.8/site-packages"
	platstdlib = "/tmp/build-env-kit8jz7e/lib/python3.8"
	purelib = "/tmp/build-env-kit8jz7e/lib/python3.8/site-packages"
	scripts = "/tmp/build-env-kit8jz7e/bin"
	stdlib = "/tmp/build-env-kit8jz7e/lib/python3.8"

(on a virtual environment, pyvenv.cfg is found)

$ PYTHONHOME=/tmp/build-env-kit8jz7e PYTHONPATH=/usr/lib/python38.zip:/usr/lib/python3.8:/usr/lib/python3.8/
lib-dynload:/usr/lib/python3.8/site-packages:/tmp/build-env-kit8jz7e/lib/python3.8/site-packages:/tmp/build-env-kit8jz7e/lib/python3.8/site-packages ~/.virtualenvs/test-env/bin/python -m sysconfig | head -n 15
Platform: "linux-x86_64"
Python version: "3.8"
Current installation scheme: "posix_prefix"

Paths:
	data = "/home/anubis/.virtualenvs/test-env"
	include = "/tmp/build-env-kit8jz7e/include/python3.8"
	platinclude = "/tmp/build-env-kit8jz7e/include/python3.8"
	platlib = "/home/anubis/.virtualenvs/test-env/lib/python3.8/site-packages"
	platstdlib = "/home/anubis/.virtualenvs/test-env/lib/python3.8"
	purelib = "/home/anubis/.virtualenvs/test-env/lib/python3.8/site-packages"
	scripts = "/home/anubis/.virtualenvs/test-env/bin"
	stdlib = "/tmp/build-env-kit8jz7e/lib/python3.8"

So, if pyvenv.cfg is present, even if I provide a PYTHONHOME, which should relocate the Python installation to other path, the paths gets overridden. PYTHONHOME should override the prefix, but for some reason pyvenv.cfg gets precedence, making PYTHONHOME useless.

With PEP 405, the Python interpreter is not self-contained, it depends on external environment variables we might not control. I chose PYTHONHOME because it did not require us to run a different interpreter, but it turns out it does. I think PEP 405, which was motivated by the lack of symlink support in Windows, in its attempt to make things better killed this use case, I wish just a little bit of more thought had been put into it.

from build.

Hum, I am pretty sure I was passing --ignore-installed to pip, maybe that got removed in one of the reviews. Can you check if it fixes your issue?

from build.

Hum, I am pretty sure I was passing --ignore-installed to pip, maybe that got removed in one of the reviews. Can you check if it fixes your issue?

That would only partially fix the issue — pip probably shouldn't need --ignore-installed because it shouldn't be seeing any packages installed in the isolated environment when it's invoked. It probably would solve the issue of the default upgrading behavior, but it won't solve the other issue I mentioned, which is that packages installed in the outer environment are visible in the isolated environment (see the example about dateutil in my earlier post here).

from build.

I will keep this short, otherwise I will rant about this. It happens because of the presence of pyvenv.cfg which overrides the PYTHONHOME behavior.

from build.

Opened pypa/pyproject-hooks#92 which is required to fix this issue.

from build.

Keeping it short. IMO PYTHONHOME should override pyvenv.cfg. PEP 405 tried to improve things by introducing pyvenv.cfg but it made PYTHONHOME unreliable. I would actually call this a breaking change, it should have never happened between minor versions. I am very bummed out from finding out about this.

from build.

I would actually call this a breaking change, it should have never happened between minor versions.

You mean PEP 405 was a breaking change? When PEPs propose a change Python semantics, they're never implemented in minor versions. Looks like PEP 405 was introduced in Python 3.3 (Python does not use semver — backwards-incompatible changes are allowed in point releases like 3.2 → 3.3).

Keeping it short. IMO PYTHONHOME should override pyvenv.cfg. PEP 405 tried to improve things by introducing pyvenv.cfg but it made PYTHONHOME unreliable.

This is a somewhat vague description of the problem. I don't really understand the isolation / environment code well enough to understand why this would be a problem. From what I can tell, PYTHONHOME refers to the location of the standard library, but the problem we're having is not from the standard library, it's from site-packages. I'll note that it's entirely possible to prevent access to these by removing the relevant directories from sys.path (though I'm not sure if it's possible to do that with environment variables).

Opened pypa/pep517#92 which is required to fix this issue.

One possible (very hacky) workaround in the meantime might be to monkey-patch sys.executable during the run in versions of pep517 that don't support using a custom executable.

from build.

Interesting, I wonder if it's worth tweaking the documentation to make it more clear that PYTHONHOME / pyvenv.cfg's home value is setting the base path for a whole bunch of stuff, not just the standard library. Maybe that's more obvious to people familiar with the way all those other paths are derived?

from build.

That does work, yes.

Paths:
	data = "/tmp/build-env-65qowjyv"
	include = "/tmp/build-env-65qowjyv/include/python3.8"
	platinclude = "/tmp/build-env-65qowjyv/include/python3.8"
	platlib = "/tmp/build-env-65qowjyv/lib/python3.8/site-packages"
	platstdlib = "/tmp/build-env-65qowjyv/lib/python3.8"
	purelib = "/tmp/build-env-65qowjyv/lib/python3.8/site-packages"
	scripts = "/tmp/build-env-65qowjyv/bin"
	stdlib = "/tmp/build-env-65qowjyv/lib/python3.8"

from build.

@pganssle can you confirm that the current master fixes the issue for you?

from build.

Ping. Should this be closed now?

from build.

I think so.

from build.

Okay, we can reopen if needed.

from build.