saltstack / relenv Goto Github PK
View Code? Open in Web Editor NEWRe-producible and Re-relocatable Python Environments
License: Apache License 2.0
Re-producible and Re-relocatable Python Environments
License: Apache License 2.0
Recent versions of relenv (at least 0.15.0, 0.15.1, and 0.16.0) break pip install
when used with the --target
argument.
This causes salt-pip
not to work correctly when trying to install some Python modules like mysqlclient
(saltstack/salt#65980) or M2Crypto
(saltstack/salt#66311).
I can definitely track this down to an issue in relenv
because the problem can be resolved by commenting removing a single line in relenvβs runtime.py
:
--- relenv/runtime.py.orig 2024-05-17 14:49:35.522975150 +0000
+++ relenv/runtime.py 2024-05-17 14:51:00.735990855 +0000
@@ -783,7 +783,6 @@
Wrapper("pip._internal.operations.install.wheel", wrap_pip_install_wheel),
Wrapper("pip._internal.operations.install.legacy", wrap_pip_install_legacy),
Wrapper("pip._internal.operations.build.wheel", wrap_pip_build_wheel),
- Wrapper("pip._internal.commands.install", wrap_cmd_install),
Wrapper("pip._internal.locations", wrap_locations),
Wrapper("pip._internal.cli.req_command", wrap_req_command),
Wrapper("pip._internal.req.req_install", wrap_req_install),
As this might have undesired side effects (I do not understand the code well enough to be sure): I also tried more localized changes and came up with the following patch that resolves the issue:
--- relenv/runtime.py.orig 2024-05-17 14:49:35.522975150 +0000
+++ relenv/runtime.py 2024-05-17 14:53:50.545215677 +0000
@@ -641,19 +641,6 @@
mod.InstallCommand.run = wrap(mod.InstallCommand.run)
- def wrap(func):
- @functools.wraps(func)
- def wrapper(self, target_dir, target_temp_dir, upgrade):
- from pip._internal.cli.status_codes import SUCCESS
-
- return SUCCESS
-
- return wrapper
-
- if hasattr(mod.InstallCommand, "_handle_target_dir"):
- mod.InstallCommand._handle_target_dir = wrap(
- mod.InstallCommand._handle_target_dir
- )
return mod
@@ -670,8 +657,6 @@
):
scheme = func(dist_name, user, home, root, isolated, prefix)
if TARGET.TARGET:
- scheme.platlib = TARGET.PATH
- scheme.purelib = TARGET.PATH
scheme.data = TARGET.PATH
return scheme
@@ -729,8 +714,6 @@
use_user_site=False,
pycompile=True,
):
- if TARGET.TARGET:
- home = TARGET.PATH
return func(
self,
install_options,
@@ -756,8 +739,6 @@
use_user_site=False,
pycompile=True,
):
- if TARGET.TARGET:
- home = TARGET.PATH
return func(
self,
global_options,
In order to resolve the error when trying to install the module, the change that avoids setting scheme.platlib
and scheme.purelib
is sufficient, but when only applying this change and not the other changes, the installed modules end up in /opt/saltstack/salt/extras-3.10/lib/python
instead of /opt/saltstack/salt/extras-3.10
.
When applying the change regarding scheme.purelib
and home = TARGET.PATH
, but not removing the wrapper for _handle_target_dir
, there is no error but the module is not installed at all.
If I understand it correctly, home
is usually set to a temporary directory and _handle_target_dir
copies it from there to the actual target path. This is why not overriding home
but replacing _handle_target_dir
results in the module not being installed at all (it ends up in the temporary directory and is never moded from there).
@dwoz Unfortunately, I do not understand the intention behind this code, so I am not sure whether my fix is correct or might break something else. As it seems like you wrote most of this code, I hope that you might be able to explain what this code is supposed to do.
Maybe this code was targeted at an older version of pip, and due to changes in pip, it does not work correctly any longer (after all, the code messes with internals of pip). I tested my patch with Salt 3006.8, which bundles pip 23.3.2.
Some of the commits that added the code that is removed by my patch say something about fixing pip uninstall
, but at fore the aforementioned version of pip, uninstall still works correctly after applying my patch.
Defining the RELENV_PIP_DIR environment variable should cause pip to install scripts into the root the environment instead of the bin
directory. This functionality works on linux but not on windows. I suspect this has to do with the version of python on windows and not something platform specific.
When I try to run pip commands I get the following:
Salts-Mac:osx saltadmin$ /Users/saltadmin/src/salt/pkg/osx/build/opt/salt/bin/pip3 list
readlink: illegal option -- f
usage: readlink [-n] [file ...]
usage: dirname path
/Users/saltadmin/src/salt/pkg/osx/build/opt/salt/bin/pip3: line 2: /python3: No such file or directory
/Users/saltadmin/src/salt/pkg/osx/build/opt/salt/bin/pip3: line 2: exec: /python3: cannot execute: No such file or directory
Running pip from python works:
Salts-Mac:osx saltadmin$ /Users/saltadmin/src/salt/pkg/osx/build/opt/salt/bin/python3 -m pip list
Package Version
---------- -------
pip 22.3.1
relenv 0.4.9
setuptools 65.5.0
wheel 0.38.4
Installing packages with salt-pip
(Salt 3006.0) fails with a JSONDecodeError
and what appears to be some recursive behavior.
To reproduce, set up a fresh Almalinux 8.7 box (I use Vagrant on QEMU) and manually install the Salt 3006.0 minion. Add a symlink according to #114:
ln -s /opt/saltstack/salt/bin/python3 /usr/bin/python3
Then, attempt to install pycryptodome
(or any other package in my experience):
/opt/saltstack/salt/salt-pip install pycryptodome
Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/saltstack/salt/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/__main__.py", line 29, in <module>
from pip._internal.cli.main import main as _main
File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_internal/cli/main.py", line 9, in <module>
from pip._internal.cli.autocompletion import autocomplete
File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_internal/cli/autocompletion.py", line 10, in <module>
from pip._internal.cli.main_parser import create_main_parser
File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_internal/cli/main_parser.py", line 9, in <module>
from pip._internal.build_env import get_runnable_pip
File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_internal/build_env.py", line 20, in <module>
from pip._internal.cli.spinners import open_spinner
File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_internal/cli/spinners.py", line 9, in <module>
from pip._internal.utils.logging import get_indentation
File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_internal/utils/logging.py", line 29, in <module>
from pip._internal.utils.misc import ensure_dir
File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_internal/utils/misc.py", line 42, in <module>
from pip._internal.locations import get_major_minor_version
File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_internal/locations/__init__.py", line 14, in <module>
from . import _sysconfig
File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_internal/locations/_sysconfig.py", line 11, in <module>
from .base import change_root, get_major_minor_version, is_osx_framework
File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_internal/locations/base.py", line 16, in <module>
site_packages: typing.Optional[str] = sysconfig.get_path("purelib")
File "/opt/saltstack/salt/lib/python3.10/sysconfig.py", line 572, in get_path
return get_paths(scheme, vars, expand)[name]
File "/opt/saltstack/salt/lib/python3.10/site-packages/relenv/runtime.py", line 167, in wrapped
paths = func(scheme=scheme, vars=vars, expand=expand)
File "/opt/saltstack/salt/lib/python3.10/sysconfig.py", line 562, in get_paths
return _expand_vars(scheme, vars)
File "/opt/saltstack/salt/lib/python3.10/sysconfig.py", line 218, in _expand_vars
_extend_dict(vars, get_config_vars())
File "/opt/saltstack/salt/lib/python3.10/site-packages/relenv/runtime.py", line 131, in wrapped
_CONFIG_VARS = func()
File "/opt/saltstack/salt/lib/python3.10/sysconfig.py", line 640, in get_config_vars
srcdir = os.path.dirname(get_makefile_filename())
File "/opt/saltstack/salt/lib/python3.10/sysconfig.py", line 399, in get_makefile_filename
return os.path.join(get_path('stdlib'), config_dir_name, 'Makefile')
File "/opt/saltstack/salt/lib/python3.10/sysconfig.py", line 572, in get_path
return get_paths(scheme, vars, expand)[name]
File "/opt/saltstack/salt/lib/python3.10/site-packages/relenv/runtime.py", line 167, in wrapped
paths = func(scheme=scheme, vars=vars, expand=expand)
File "/opt/saltstack/salt/lib/python3.10/sysconfig.py", line 562, in get_paths
return _expand_vars(scheme, vars)
File "/opt/saltstack/salt/lib/python3.10/sysconfig.py", line 218, in _expand_vars
_extend_dict(vars, get_config_vars())
File "/opt/saltstack/salt/lib/python3.10/site-packages/relenv/runtime.py", line 140, in wrapped
_SYSTEM_CONFIG_VARS = json.loads(p.stdout[:-1])
File "/opt/saltstack/salt/lib/python3.10/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/opt/saltstack/salt/lib/python3.10/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/opt/saltstack/salt/lib/python3.10/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The JSONDecodeError
is due to get_config_vars_wrapper(...)
parsing the stdout of a subprocess Python call to sysconfig: in my environment, calling sysconfig.get_config_vars()
actually prints nothing (stdout empty), causing JSON decoding to fail. (Side note: did Rube Goldberg write this function? I'm sure there's a reason for the way this is implemented; could someone elaborate?)
Digging deeper into sysconfig.get_config_vars()
reveals some apparently recursive behavior. The following is what I have observed in my environment when calling sysconfig.get_config_vars()
in the same manner as get_config_vars_wrapper()
from Salt 3006.0's Python 3.10 REPL:
sysconfig.get_config_vars()
is called without arguments. _CONFIG_VARS
is None
, the os.name
is posix
and _PYTHON_BUILD
is falsy. Execution continues to get_makefile_filename()
.get_path('stdlib')
.get_paths('posix_prefix', None, True)
is called. It in turn calls _expand_vars('posix_prefix', None)
._expand_vars
calls _extend_dict(None, get_config_vars())
. Notice that get_config_vars()
is now being recursively called.get_config_vars
. The wrapper, again, calls sysconfig.get_config_vars(None)
within a new Python subprocess, and this time _CONFIG_VARS
has a bunch (722 for me) of variables, which it returns._extend_dict
. In fact, _extend_dict
is never called. Instead, the wrapper recursively calls itself. I've seen it do so up to 1,024 times, and have heard from another user with this problem that it is accompanied by a spike in memory and crash of the Salt minion and master.Interestingly, this behavior seems to only occur when using Salt's bundled Python at /usr/bin/python3
. Compiling Python 3.10.11 from source and adding the appropriate symlink for that scenario (ln -s /usr/local/bin/python3 /usr/bin/python3
) does not present this problem.
Firstly, I've been watching relenv for a while - I think it's rad. Great work π
I would like to add support for customizing the build environment (e.g. what's created with relenv build
). Specifically:
[Cc]onfigure
flags for CPython, OpenSSL, SQLite, etc.CFLAGS
/ LDFLAGS
/ CXXFLAGS
MACOSX_DEPLOYMENT_TARGET
)Although my primary focus is on macOS (as I support my Company's fleet of macOS devices, including its use of Salt), I think customization would be valuable for all platforms.
As a good example, take issue #114, where the default /usr/bin/python3
executable is hard-coded in relenv, and is actually immutable on macOS (due to System Integrity Protection), but can "point" to different Xcode environments (via xcode-select
).
If I wanted to use zlib
or libzmq
from Homebrew (instead of building it myself), currently relenv doesn't allow me to do that.
I want to be able use a different Python version (e.g. a pre-release version, 3.9.x, or any other version not specifically hard-coded into relenv
.
Would be nice to be able to suppress the output if needed
When attempting to install salt using the onedir method I encountered a failure where some imports in the relenv Python environment could not be found (ex. runpy). After doing some digging, I was able to identify that the mechanism relenv uses to sanitize sys.path
will remove valid entries if the Python interpreter is called from a path that contains a symlink anywhere along the path.
mkdir -p target/ && ln -sf target/ parent/
.cd parent/ && relenv create testenv
.sys.path
has been cleared.$ RELENV_DEBUG=true parent/testenv/bin/python3
Unable to get the modules directory from openssl: version: Option unknown option -m
version: Use -help for summary.
original sys.path was ['/root/parent/testenv/lib/python310.zip', '/root/parent/testenv/lib/python3.10', '/root/parent/testenv/lib/python3.10/lib-dynload', '/root/parent/testenv/lib/python3.10/site-packages']
new sys.path is []
After site customize wrapper
Python 3.10.13 (main, Nov 8 2023, 12:14:32) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
After digging into the code, I identified that the sanitize_sys_path
function incorrectly removes entries in sys.path
because it doesn't resolve symlinks properly. Specifically this code:
A call to resolve()
needs to happen before checking if the sys.path
entry is relative to the valid prefix. Something like the following would resolve this issue.
__resolved_path = pathlib.Path(__path).resolve()
__resolved_path.relative_to(__valid_path_prefix)
__sys_path.append(str(__resolved_path))
Currently the lack of proper path resolving breaks installing salt in onedir mode if /opt
is symlinked to another directory.
Looks like docs have not been updated during the last few releases. We need to investigate why docs were not publish and If docs fail the build should fail.
Currently we're cross compiling python aarch64 for linux on x86_64.
Add a build which builds aarch64 on aarch64 to make sure we're testing this case for now.
We need to be able to build 32-bit Python on Windows. As per the output of --help
the only options are x86_64
and aarch64
:
C:\src\Mayflower>python -m mayflower build --help
usage: __main__.py build [-h] [--arch {x86_64,aarch64}] [--clean] [--no-cleanup] [--no-download] [--step STEP]
Build Mayflower Python Environments from source
optional arguments:
-h, --help show this help message and exit
--arch {x86_64,aarch64}
The host architecture [default: x86_64]
--clean Clean up before running the build. This option will remove the logs, src, build, and previous tarball.
--no-cleanup By default the build directory is removed after the build tarball is created. Setting this option will leave the build directory in place.
--no-download Skip downloading source tarballs
--step STEP A step to run alone, can use multiple of this argument. When this option is used to invoke builds, depenencies of the steps are ignored. This option should be used with care, as it's easy to request a situation that has no chance of
being succesful.```
The c-extensions built by newer cryptography versions are not getting the proper rpath set.
Looking through the long list of bundled dependencies, I'm slightly concerned about how old some of the software releases are, especially given that the whole point of this tool is to have zero dependencies. Some are somewhat unimportant/irrelevant, but others like libssl and python are frankly quite important
Looking through the bundled software (for Linux at least; it's the only platform that interests me) here https://github.com/saltstack/relative-environment-for-python/blob/b848c6f7cf7a36587935db349c42e89e0fb0e5b6/relenv/build/linux.py#L353
Package | Relenv version | Latest version | Notes |
---|---|---|---|
glibc | 2.17 | 2.36 | This was released 2012-12-25 so is over 10 years old. I don't think there's really any excuse for this one |
libssl | 1.1.1q | 3.0.7 (or at least the latest security patch: 1.1.1s) | OpenSSL is security software, so by definition being behind means it's vulnerable. There's also no reason to not use 3.x here; 1.1.x is not going to be supported forever |
python | 3.10.6 | 3.11.1 | 3.11 has been out only 13 weeks, but there's strong evidence that it has significant performance gains, especially for Salt in some of the ..less performant parts of the code. I've seen 3.11 drop minutes off some highstate times |
zlib | 1.12.12 | 1.12.13 | 1.12.13 fixes a couple of CVEs in 1.12.12. Not critical, but still security related |
From what I can see, this doesn't even have the excuse of "it was latest when we started" because many of these "latest" versions even predate the initial commit of this data: b48d9ca
I would consider these issues above significant enough to not consider using relenv-based Salt packages until these have been addressed. To me, the whole (and only) reason to go to all of the effort to build what is effectively an isolated system of libraries (like you would in a container) like this is to be able to break away from the grasp of "stable" distributions with old versions and to be able to use new features, bug/security fixes and performance enhancements. Instead, what relenv provides currently is a liability to fork, build and maintain newer versions of libraries to stay secure, but no perceivable benefits for the user
In the following example scripts will be missing.
myrelenv/bin/pip install --target=myrelenv/extras rend
myrelenv/bin/pip install --target=myrelenv/extras cowsay
The cowsay script is missing from myrelenv/extras/bin
See also saltstack/salt#64662
$ relenv --version
0.11.2
$ relenv create --help
Traceback (most recent call last):
File "/Users/rick.bowen/git/saltstack/relative-environment-for-python/.venv/bin/relenv", line 8, in <module>
sys.exit(main())
File "/Users/rick.bowen/git/saltstack/relative-environment-for-python/relenv/__main__.py", line 49, in main
args = parser.parse_args()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 1826, in parse_args
args, argv = self.parse_known_args(args, namespace)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 1859, in parse_known_args
namespace, args = self._parse_known_args(args, namespace)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 2050, in _parse_known_args
positionals_end_index = consume_positionals(start_index)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 2027, in consume_positionals
take_action(action, args)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 1936, in take_action
action(self, namespace, argument_values, option_string)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 1214, in __call__
subnamespace, arg_strings = parser.parse_known_args(arg_strings, None)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 1859, in parse_known_args
namespace, args = self._parse_known_args(args, namespace)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 2068, in _parse_known_args
start_index = consume_optional(start_index)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 2008, in consume_optional
take_action(action, args, option_string)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 1936, in take_action
action(self, namespace, argument_values, option_string)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 1099, in __call__
parser.print_help()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 2556, in print_help
self._print_message(self.format_help(), file)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 2540, in format_help
return formatter.format_help()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 283, in format_help
help = self._root_section.format_help()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 214, in format_help
item_help = join([func(*args) for func, args in self.items])
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 214, in <listcomp>
item_help = join([func(*args) for func, args in self.items])
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 501, in _format_text
return self._fill_text(text, text_width, indent) + '\n\n'
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 650, in _fill_text
text = self._whitespace_matcher.sub(' ', text).strip()
TypeError: expected string or bytes-like object
There should be a way to detect if the python being used is a relenv onedir or not.
Checking if relenv is installed is not enough. Relenv can be installed into python environments to bootstrap onedir environments, in fact, that's how relenv works.
So, how can we detect?
Python has built-in support for frozen(sys.frozen
). PyInstaller uses sys.MEIPASS
.
Set sys.relenv = True
when relenv is bootstrapped at runtime?
When using pip with --target
the shebangs in python scripts created by pip are broken.
Since about 20 June, some tests and installations of Ubuntu machines involving the pip.installed command have been failing. The problem's appearance coincided with the appearance of pip version 24.1, but the affected Ubuntu machines have version 22.0.2. The affected Ubuntu machines are using relenv version 0.16.0, Python 3.10.14 and salt-minion 3006.8.
The error appears in this way:
ID: pip (This part is okay)
Function: pip.installed
Result: True
Comment: All packages were successfully installed
Started: 11:48:07.543580
Duration: 9436.212 ms
Changes:
----------
pip==24.1:
InstalledID: pyinotify
Function: pip.installed
Result: False
Comment: Failed to install packages: pyinotify. Error: Looking in indexes: (index removed)
Collecting pyinotify
Downloading (URL removed) (60 kB)
>Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½Γ’βοΏ½ 61.0/61.0 kB 1.0 MB/s eta 0:00:00
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
Building wheels for collected packages: pyinotify
Building wheel for pyinotify (setup.py): started
Building wheel for pyinotify (setup.py): finished with status 'done'
Created wheel for pyinotify: filename=pyinotify-0.9.6-py3-none-any.whl size=25342 sha256=2ed656274ed34a4ddf69abdcf7d167729fe972c03b6b4d900e5e1bb4fa8394b7
Stored in directory: /root/.cache/pip/wheels/4a/04/c7/8ad5f6b9280b1fe87df10fd6473716f4f1f6a922fbbfb94bc8
Successfully built pyinotify
Installing collected packages: pyinotify ERROR: Exception:
Traceback (most recent call last):
File "/opt/saltstack/salt/extras-3.10/pip/_internal/cli/base_command.py", line 179, in exc_logging_wrapper
status = run_func(*args)
File "/opt/saltstack/salt/lib/python3.10/site-packages/relenv/runtime.py", line 638, in wrapper
return func(self, options, args)
File "/opt/saltstack/salt/extras-3.10/pip/_internal/cli/req_command.py", line 67, in wrapper
return func(self, options, args)
File "/opt/saltstack/salt/extras-3.10/pip/_internal/commands/install.py", line 455, in run
installed = install_given_reqs(
File "/opt/saltstack/salt/extras-3.10/pip/_internal/req/init.py", line 70, in install_given_reqs
requirement.install(
File "/opt/saltstack/salt/lib/python3.10/site-packages/relenv/runtime.py", line 761, in wrapper
return func(
File "/opt/saltstack/salt/extras-3.10/pip/_internal/req/req_install.py", line 817, in install
scheme = get_scheme(
File "/opt/saltstack/salt/lib/python3.10/site-packages/relenv/runtime.py", line 673, in wrapper
scheme.platlib = TARGET.PATH
File "", line 4, in setattr
dataclasses.FrozenInstanceError: cannot assign to field 'platlib'
Started: 11:48:17.530499
Duration: 2453.652 ms
Changes:
As a result, inotify beacons cannot be configured. This apparently affects at least one other module in pip.install commands. So far, this has only been observed in tests simulating the installation (i.e. initial highstate) of Ubuntu machines and in Ubuntu machines installed after 20 June. Machines installed at or before this date seem unaffected, even when they also have relenv version 0.16.0, Python 3.10.14 as well as salt-minion 3006.8. It is known to affect Ubuntu 20.04 and 22.04. Earlier and later versions of Ubuntu were not tested.
Deleting line 673 of /opt/saltstack/salt/lib/python3.10/site-packages/relenv/runtime.py resulted in the dataclasses.FrozenInstanceError appearing for 'purelib' instead (line 674). Deleting this line too moved the dataclasses.FrozenInstanceError to 'data' (line 675). Deleting line 675 and the if statement in line 672 removed the error messages, but this naturally does not install the module.
Using sitecustomize.py to bootstrap relenv's runtime prevents other tools from leveraging sitecustomize.py.
The toolchains seem to have started failing to build because crosstool-ng is un-able to download zlib.
We discovered this when using the pre-built macos python build to make the Salt Mac package. It seems we're linking to gettext in /usr/local for some reason. When installing salt into the environment we get the following traceback.
dyld: Library not loaded: /usr/local/opt/gettext/lib/libintl.8.dylib
Referenced from: /Users/saltadmin/src/salt/pkg/osx/build/opt/salt/bin/python3
Reason: image not found
/Users/saltadmin/src/salt/pkg/osx/install_salt.sh: line 102: 968 Abort trap: 6
Everything works when using relenv build to build the python on the macos box where this was tested.
Currently, the version of Python built by Mayflower is hard-coded. We need the ability to pass a version of Python to build.
No virtualenv:
Python 3.11.0 (main, Sep 12 2023, 18:26:57) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hashlib
>>> hashlib.md5()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: [digital envelope routines] unsupported
>>> hashlib.sha256()
<sha256 _hashlib.HASH object @ 0x7f030577b7d0>
>>>
Virtualenv created using the system python:
root@ip-10-2-3-37 [ ~ ]# python3 -m virtualenv /tmp/ve1
created virtual environment CPython3.11.0.final.0-64 in 346ms
creator CPython3Posix(dest=/tmp/ve1, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
added seed packages: pip==23.2.1, setuptools==68.2.2, wheel==0.41.2
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
root@ip-10-2-3-37 [ ~ ]# source /tmp/ve1/bin/activate
(ve1) root@ip-10-2-3-37 [ ~ ]# python
Python 3.11.0 (main, Sep 12 2023, 18:26:57) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hashlib
>>> hashlib.md5()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: [digital envelope routines] unsupported
>>> hashlib.sha256()
<sha256 _hashlib.HASH object @ 0x7f4c324e6670>
>>>
No virtualenv
root@ip-10-2-3-37 [ ~ ]# /tmp/testing/artifacts/salt/bin/python3
Python 3.10.13 (main, Sep 9 2023, 07:21:14) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hashlib
>>> hashlib.md5()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: [digital envelope routines] unsupported
>>> hashlib.sha256()
<sha256 _hashlib.HASH object @ 0x7f0b18d01d50>
>>>
A virtualenv using relenv
root@ip-10-2-3-37 [ ~ ]# /tmp/testing/artifacts/salt/bin/python3 -m pip install virtualenv
Collecting virtualenv
Downloading virtualenv-20.24.5-py3-none-any.whl (3.7 MB)
ββββββββββββββββββββββββββββββββββββββββ 3.7/3.7 MB 14.6 MB/s eta 0:00:00
Collecting filelock<4,>=3.12.2
Downloading filelock-3.12.4-py3-none-any.whl (11 kB)
Collecting platformdirs<4,>=3.9.1
Downloading platformdirs-3.11.0-py3-none-any.whl (17 kB)
Collecting distlib<1,>=0.3.7
Downloading distlib-0.3.7-py2.py3-none-any.whl (468 kB)
ββββββββββββββββββββββββββββββββββββββββ 468.9/468.9 kB 30.3 MB/s eta 0:00:00
Installing collected packages: distlib, platformdirs, filelock, virtualenv
Successfully installed distlib-0.3.7 filelock-3.12.4 platformdirs-3.11.0 virtualenv-20.24.5
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
[notice] A new release of pip available: 22.3.1 -> 23.3.1
[notice] To update, run: /tmp/testing/artifacts/salt/bin/python3 -m pip install --upgrade pip
root@ip-10-2-3-37 [ ~ ]# /tmp/testing/artifacts/salt/bin/python3 -m virtualenv /tmp/ve2
created virtual environment CPython3.10.13.final.0-64 in 314ms
creator CPython3Posix(dest=/tmp/ve2, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
added seed packages: pip==23.2.1, setuptools==68.2.2, wheel==0.41.2
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
root@ip-10-2-3-37 [ ~ ]# source /tmp/ve2/bin/activate
(ve2) root@ip-10-2-3-37 [ ~ ]# python
Python 3.10.13 (main, Sep 9 2023, 07:21:14) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hashlib
>>> hashlib.md5()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: [digital envelope routines] unsupported
>>> hashlib.sha256()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: [digital envelope routines] unsupported
>>>
Refs saltstack/salt#65444
Add two more types of keys. In addition to minion; add master and client (cli) keys.
Installing Pycurl needs a libcurl that is compiled against the proper glibc version. Document the process of building libcurl to work with relenv.
<frozen importlib._bootstrap>:914: ImportWarning: RelenvImporter.find_spec() not found; falling back to find_module()
A couple warnings started popping up.
2023-08-28T21:30:28.2106254Z <frozen importlib._bootstrap>:283: DeprecationWarning: the load_module() method is deprecated and slated for removal in Python 3.12; use exec_module() instead
2023-08-28T21:30:28.8675492Z <frozen importlib._bootstrap>:914: ImportWarning: RelenvImporter.find_spec() not found; falling back to find_module()
get_config_vars_wrapper(...)
runs Python as a subprocess (more explanation on why this is necessary would be great). The path to Python to run in the subprocess is hard coded to /usr/bin/python3
, which is not always installed and would require either a symlink to Salt's salt-pip
or a system Python installation.
To reproduce, spin up Almalinux 8.7 (I do so in Vagrant on QEMU) and manually install the Salt minion, confirming there is no Python installed on the system. Then, log in and execute:
/opt/saltstack/salt/salt-pip install pycryptodome
Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/saltstack/salt/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/__main__.py", line 29, in <module>
from pip._internal.cli.main import main as _main
File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_internal/cli/main.py", line 9, in <module>
from pip._internal.cli.autocompletion import autocomplete
File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_internal/cli/autocompletion.py", line 10, in <module>
from pip._internal.cli.main_parser import create_main_parser
File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_internal/cli/main_parser.py", line 9, in <module>
from pip._internal.build_env import get_runnable_pip
File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_internal/build_env.py", line 20, in <module>
from pip._internal.cli.spinners import open_spinner
File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_internal/cli/spinners.py", line 9, in <module>
from pip._internal.utils.logging import get_indentation
File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_internal/utils/logging.py", line 29, in <module>
from pip._internal.utils.misc import ensure_dir
File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_internal/utils/misc.py", line 42, in <module>
from pip._internal.locations import get_major_minor_version
File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_internal/locations/__init__.py", line 14, in <module>
from . import _sysconfig
File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_internal/locations/_sysconfig.py", line 11, in <module>
from .base import change_root, get_major_minor_version, is_osx_framework
File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_internal/locations/base.py", line 16, in <module>
site_packages: typing.Optional[str] = sysconfig.get_path("purelib")
File "/opt/saltstack/salt/lib/python3.10/sysconfig.py", line 572, in get_path
return get_paths(scheme, vars, expand)[name]
File "/opt/saltstack/salt/lib/python3.10/site-packages/relenv/runtime.py", line 167, in wrapped
paths = func(scheme=scheme, vars=vars, expand=expand)
File "/opt/saltstack/salt/lib/python3.10/sysconfig.py", line 562, in get_paths
return _expand_vars(scheme, vars)
File "/opt/saltstack/salt/lib/python3.10/sysconfig.py", line 218, in _expand_vars
_extend_dict(vars, get_config_vars())
File "/opt/saltstack/salt/lib/python3.10/site-packages/relenv/runtime.py", line 131, in wrapped
_CONFIG_VARS = func()
File "/opt/saltstack/salt/lib/python3.10/sysconfig.py", line 640, in get_config_vars
srcdir = os.path.dirname(get_makefile_filename())
File "/opt/saltstack/salt/lib/python3.10/sysconfig.py", line 399, in get_makefile_filename
return os.path.join(get_path('stdlib'), config_dir_name, 'Makefile')
File "/opt/saltstack/salt/lib/python3.10/sysconfig.py", line 572, in get_path
return get_paths(scheme, vars, expand)[name]
File "/opt/saltstack/salt/lib/python3.10/site-packages/relenv/runtime.py", line 167, in wrapped
paths = func(scheme=scheme, vars=vars, expand=expand)
File "/opt/saltstack/salt/lib/python3.10/sysconfig.py", line 562, in get_paths
return _expand_vars(scheme, vars)
File "/opt/saltstack/salt/lib/python3.10/sysconfig.py", line 218, in _expand_vars
_extend_dict(vars, get_config_vars())
File "/opt/saltstack/salt/lib/python3.10/site-packages/relenv/runtime.py", line 132, in wrapped
p = subprocess.run(
File "/opt/saltstack/salt/lib/python3.10/subprocess.py", line 503, in run
with Popen(*popenargs, **kwargs) as process:
File "/opt/saltstack/salt/lib/python3.10/subprocess.py", line 971, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/opt/saltstack/salt/lib/python3.10/subprocess.py", line 1863, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/bin/python3'
This issue can be bypassed by:
ln -s /opt/saltstack/salt/bin/python3 /usr/bin/python3
There should be better documentation about all of the things relenv does when python starts up.
Upgrade openssl to account for recent CVEs. Upgrade python versions. Also tackle any other dependencies that can be upgrades
Did relenv fetch, then relenv create. Result:
β ~ myenv/bin/python3 -V
myenv/bin/python3: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory
We should validate the shebangs generated by relenv using the shellcheck tool.
We need to add a step to the workflows to upload our toolchains and builds to repo.saltproject.io.
Salt 3006.5, Ubuntu 22.04:
salt-pip install debugpy
/opt/saltstack/salt/bin/python3 -c 'import debugpy; debugpy.listen(5678); debugpy.wait_for_client()'
0.69s - Error patching args (debugger not attached to subprocess).
Traceback (most recent call last):
File "/opt/saltstack/salt/extras-3.10/debugpy/_vendored/pydevd/_pydev_bundle/pydev_monkey.py", line 526, in patch_args
host, port = _get_host_port()
File "/opt/saltstack/salt/extras-3.10/debugpy/_vendored/pydevd/_pydev_bundle/pydev_monkey.py", line 212, in _get_host_port
host, port = pydevd.dispatch()
File "/opt/saltstack/salt/extras-3.10/debugpy/_vendored/pydevd/pydevd.py", line 3084, in dispatch
host = setup['client']
TypeError: 'NoneType' object is not subscriptable
The same command works with system Python binary.
relocate is not setting rpath properly on cross compiled builds. The root of the problem seems to be the cross toolchain's ldd is not returning the expected output.
Building M2Crypto for a relenv environment results in a segfault on import.
Can we choose PYTHONUTF8=1
behavior to be the default on windows? This is when building python.
https://relenv.readthedocs.io/en/latest/toolchain.html references [email protected]:saltstack/relenv.git
(by ssh, not https, too) which doesn't exist, and tells the user to use relenv toolchain build
which doesn't exist:
(venv) frebib@frebib-PC /t/relenv.NCsd7Umb> python3.11 -m relenv toolchain download
usage: relenv toolchain [-h] [--arch {x86_64,aarch64}] [--clean] [--crosstool-only] {build,fetch}
relenv toolchain: error: argument command: invalid choice: 'download' (choose from 'build', 'fetch')
It would be nice to have the option to see the debug output to make debugging easier
Hashutils imported before ssl modules directory has been detected. The result is missing hashes when trying to use Openssl's legacy provider.
The 3006.2 release of Salt introduced a new version of relenv, which added code to runtime.py that sets OPENSSL_MODULES
. It also added openssl modules to the relenv. However, this transparent environment mangling breaks (at least some) non-Salt tooling that uses OpenSSL.
For example, when using Salt to manage maas, we utilize a cmd.script
state to run several shell commands that initialize a new maas instance, configure TLS, etc. With the latest changes to relenv released with Salt 3006.2, OPENSSL_MODULES
is pointed at the OpenSSL within the relenv, and any commands that Salt runs inherit this modified environment. This means that, when Salt runs maas CLI commands, the maas CLI tools try (and fail) to load legacy.so
, resulting in a cryptic OpenSSL error.
I was able to work around this by manually unsetting OPENSSL_MODULES
in the shell script, but other tools are likely to be similarly affected.
Installing salt using USE_STATIC_REQUIREMENTS=1
, which selects a different set of requirements, fails.
ubuntu@ip-10-1-102-50:~$ salt/bin/python3 -m pip install downloaded-requirements/cffi-1.14.6.tar.gz
Processing ./downloaded-requirements/cffi-1.14.6.tar.gz
Preparing metadata (setup.py) ... done
Requirement already satisfied: pycparser in ./salt/lib/python3.10/site-packages (from cffi==1.14.6) (2.21)
Building wheels for collected packages: cffi
Building wheel for cffi (setup.py) ... error
error: subprocess-exited-with-error
Γ python setup.py bdist_wheel did not run successfully.
β exit code: 1
β°β> [38 lines of output]
/home/ubuntu/salt/lib/python3.10/site-packages/setuptools/config/setupcfg.py:463: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
warnings.warn(msg, warning_class)
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-310
creating build/lib.linux-x86_64-cpython-310/cffi
copying cffi/recompiler.py -> build/lib.linux-x86_64-cpython-310/cffi
copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-cpython-310/cffi
copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-cpython-310/cffi
copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-cpython-310/cffi
copying cffi/error.py -> build/lib.linux-x86_64-cpython-310/cffi
copying cffi/pkgconfig.py -> build/lib.linux-x86_64-cpython-310/cffi
copying cffi/api.py -> build/lib.linux-x86_64-cpython-310/cffi
copying cffi/commontypes.py -> build/lib.linux-x86_64-cpython-310/cffi
copying cffi/model.py -> build/lib.linux-x86_64-cpython-310/cffi
copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-cpython-310/cffi
copying cffi/vengine_gen.py -> build/lib.linux-x86_64-cpython-310/cffi
copying cffi/ffiplatform.py -> build/lib.linux-x86_64-cpython-310/cffi
copying cffi/verifier.py -> build/lib.linux-x86_64-cpython-310/cffi
copying cffi/__init__.py -> build/lib.linux-x86_64-cpython-310/cffi
copying cffi/cparser.py -> build/lib.linux-x86_64-cpython-310/cffi
copying cffi/lock.py -> build/lib.linux-x86_64-cpython-310/cffi
copying cffi/_cffi_include.h -> build/lib.linux-x86_64-cpython-310/cffi
copying cffi/parse_c_type.h -> build/lib.linux-x86_64-cpython-310/cffi
copying cffi/_embedding.h -> build/lib.linux-x86_64-cpython-310/cffi
copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-cpython-310/cffi
running build_ext
building '_cffi_backend' extension
creating build/temp.linux-x86_64-cpython-310
creating build/temp.linux-x86_64-cpython-310/c
/home/ubuntu/.local/relenv/toolchain/x86_64-linux-gnu/bin/x86_64-linux-gnu-gcc -no-pie -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -O3 -Wall -L/home/ubuntu/.local/lib -L/home/ubuntu/.local/relenv/
toolchain/x86_64-linux-gnu/x86_64-linux-gnu/sysroot/lib -I/home/ubuntu/.local/include -I/home/ubuntu/.local/include/readline -I/home/ubuntu/.local/include/ncursesw -I/home/ubuntu/.local/relenv/toolchain/x86_64-lin
ux-gnu/x86_64-linux-gnu/sysroot/usr/include -L/home/ubuntu/.local/lib -L/home/ubuntu/.local/relenv/toolchain/x86_64-linux-gnu/x86_64-linux-gnu/sysroot/lib -I/home/ubuntu/.local/include -I/home/ubuntu/.local/includ
e/readline -I/home/ubuntu/.local/include/ncursesw -I/home/ubuntu/.local/relenv/toolchain/x86_64-linux-gnu/x86_64-linux-gnu/sysroot/usr/include -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/home/ubuntu/salt/includ
e/python3.10 -c c/_cffi_backend.c -o build/temp.linux-x86_64-cpython-310/c/_cffi_backend.o
c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory
15 | #include <ffi.h>
| ^~~~~~~
compilation terminated.
error: command '/home/ubuntu/.local/relenv/toolchain/x86_64-linux-gnu/bin/x86_64-linux-gnu-gcc' failed with exit code 1
We see the following error on photon os with fips enabled. The fips module mac between the module shipped with photon and the module shipped with relenv differ. Rather than having to run relenv with it's own ssl config; default to the system's openssl modules location.
ValueError: [digital envelope routines] unsupported
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.