Relying on per user site-packages equivalent causes problems. about s2i-python-container HOT 11 CLOSED

Blog post related to this topic and my own recommendations around when to use Python virtual environments in Docker.

• http://blog.dscpl.com.au/2016/01/python-virtual-environments-and-docker.html

from s2i-python-container.

I'm 👍 on using virtualenv / venv in the project and have raised this idea before, perhaps not as well explained as this issue describes the problem.

from s2i-python-container.

Use of a Python virtual environment could actually solve another problem. That is, it could eliminate the need to SCL enable Python in the run script or for shells using BASH_ENV and PROMPT_COMMAND.

This could be achieved by removing all existing SCL enabling code and doing the following.

1. In Dockerfile, set PATH to include /opt/app-root/venv/bin at the start.
2. In assemble, run source scl_source enable pythonXY to enable used of SCL Python version.
3. In assemble, then run virtualenv to create virtual environment at /opt/app-root/venv.

When the pip is then subsequently run, because of PATH already being set, it will pick up pip from the virtual environment just created and package will be installed into the virtual environment.

When run is later run, python will also be found via PATH from the virtual environment. It shouldn't be necessary to SCL enable Python for that to work.

This also should mean that running:

oc rsh python -V

will pick up the python from the virtual environment.

Note that #96 would still be desirable for other reasons though.

from s2i-python-container.

$ docker run --rm centos/python-27-centos7 virtualenv --version
1.10.1

$ docker run --rm openshift/python-33-centos7 virtualenv --version
1.10.1

That version of virtualenv was released on Aug 8, 2013.

$ docker run --rm centos/python-34-centos7 virtualenv --version
1.11.6

Released on May 17, 2014.

If we will have newer packaged versions of virtualenv in CentOS and RHEL via SCL I'll go ahead and make our images use a default virtual environment.

@GrahamDumpleton do you have a preference to using venv for Python versions that have it? If not, I prefer consistently using virtualenv in all images.

@hhorak @sYnfo would it be possible to get a newer version of virtualenv via SCL?

from s2i-python-container.

I would prefer to use virtualenv for all Python versions rather than using pyvenv for Python 3. There is actually a bug in pyvenv behaviour in Python 3 which means it doesn't work properly for embedded systems unless the embedded system includes a workaround for it. I have had to include a workaround in mod_wsgi so it would work anyway so it is okay, but still prefer virtualenv for all to be consistent.

BTW. If use newer virtualenv it would come with a newer pip version. So having more up to date packaged pip version may not be as big a deal. Even so pip is such an important thing that is important to keep up to date with it because of any security changes.

from s2i-python-container.

@rhcarvalho we need to track requirements for newer versions of RHSCL packages in Bugzilla. Please, report a bug with proper reasoning why it is needed and it will be considered: https://bugzilla.redhat.com/enter_bug.cgi?product=Red Hat Software Collections&component=python-virtualenv (ideally for every SCL that you care separately).

from s2i-python-container.

Just hit this problem in real world case. :-(

Requirement already satisfied (use --upgrade to upgrade): Werkzeug>=0.7 in /opt/rh/python27/root/usr/lib/python2.7/site-packages (from Flask->Lektor->-r requirements.txt (line 2))
...
Successfully installed EXIFRead Flask Lektor PyYAML argh cffi click cryptography enum34 idna inifile ipaddress itsdangerous mistune mod-wsgi ndg-httpsclient pathtools pyOpenSSL pyasn1 pycparser requests six watchdog
E0608 23:55:00.987702       1 util.go:91] + rm -rf /tmp/src
E0608 23:55:00.995379       1 util.go:91] + rm -f /opt/app-root/src/requirements.txt
E0608 23:55:01.001045       1 util.go:91] + mv /opt/app-root/src/run_lektor.py /opt/app-root/run_lektor.py
E0608 23:55:01.018828       1 util.go:91] + rm -f /opt/app-root/src/README.md
E0608 23:55:01.020415       1 util.go:91] + lektor build --output-path /opt/app-root/data
E0608 23:55:01.373292       1 util.go:91] Traceback (most recent call last):
E0608 23:55:01.373467       1 util.go:91]   File "/opt/app-root/src/.local/bin/lektor", line 7, in <module>
E0608 23:55:01.373484       1 util.go:91]     from lektor.cli import main
E0608 23:55:01.373494       1 util.go:91]   File "/opt/app-root/src/.local/lib/python2.7/site-packages/lektor/cli.py", line 9, in <module>
E0608 23:55:01.373527       1 util.go:91]     from .utils import secure_url
E0608 23:55:01.373538       1 util.go:91]   File "/opt/app-root/src/.local/lib/python2.7/site-packages/lektor/utils.py", line 24, in <module>
E0608 23:55:01.373547       1 util.go:91]     from werkzeug.urls import url_parse
E0608 23:55:01.373556       1 util.go:91] ImportError: cannot import name url_parse

The Lektor package requires a newer version than gets installed. Now to jump through hoops to workaround it.

from s2i-python-container.

Well that only got so far, then you hit problems due to babel being installed as system package.

E0609 00:08:13.747881       1 util.go:91]   File "/opt/app-root/src/.local/lib/python2.7/site-packages/lektor/types/primitives.py", line 10, in <module>
E0609 00:08:13.747886       1 util.go:91]     from babel.dates import get_timezone
E0609 00:08:13.747892       1 util.go:91] ImportError: cannot import name get_timezone

Time to give up putting hacks in requirements.txt and bootstrap in my own virtual environment to isolate it properly.

from s2i-python-container.

Alas shoving in a Python virtual environment cannot work because of the fact that pip --user was hardwired in the first place with no way to override it.

E0609 00:23:20.755124       1 util.go:91] Can not perform a '--user' install. User site-packages are not visible in this virtualenv.

from s2i-python-container.

The only way you can workaround this problem is for a user to have a custom .s2i/bin/assemble script which includes:

# Create a Python virtual environment to avoid conflicts between system
# installed Python packages and what we need. We need to activate it
# straight away so is used during build and also set up shell init
# scripts so activated for a new shell as well.

virtualenv /opt/app-root

source /opt/app-root/bin/activate

echo "source /opt/app-root/bin/activate" >> /opt/app-root/etc/scl_enable

# Upgrade pip to latest version to avoid problems due to old version.

pip install -U pip

# Invoke original assemble script to install packages. Because though it
# hardwires use of a per user site-packages directory, which conflicts
# with use of a Python virtual environment, we have to edit the assemble
# script on the fly and then execute our copy.

cat /usr/libexec/s2i/assemble | sed -e 's/ --user//' > /tmp/assemble
chmod +x /tmp/assemble
/tmp/assemble

Note that I was using /opt/app-root for other reasons, but could use /opt/app-root/venv as root for Python virtual environment.

from s2i-python-container.

Closing via #159

from s2i-python-container.