check on pkg_resources.declare_namespace breaks backwards compatibility about vsc-install HOT 10 CLOSED

the problem has nothing to do with vsc-install, it's true for anything that you try to install and that has an entry in a .pth in PYTHONPATH.
and the problem might not be easy to fix, but it can be fixed.
is this a system python? out of curiosity, if this is a system python and you would manage to actually build a vsc-base module, would loading the module actually pick the vsc-base from the module via the PYTHONPATH, or also the system one via the .pth entry for vsc-base?

from vsc-install.

using a vsc-base/2.4.2 that was already installed:

$ module load vsc-base/2.4.2
$ ls /apps/gent/SL6/sandybridge/software/vsc-base/2.4.2/lib/python2.6/site-packages/
easy-install.pth  site.py  site.pyc  vsc_base-2.4.2-py2.6.egg
$ cat /apps/gent/SL6/sandybridge/software/vsc-base/2.4.2/lib/python2.6/site-packages/easy-install.pth 
import sys; sys.__plen = len(sys.path)
./vsc_base-2.4.2-py2.6.egg
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)

$ python -c "import vsc.utils.generaloption; print vsc.utils.generaloption.__file__"
/usr/lib/python2.6/site-packages/vsc/utils/__init__.py:29: UserWarning: Module vsc was already imported from None, but /vscmnt/gent_vulpix_apps/_apps_gent/SL6/sandybridge/software/vsc-base/2.4.2/lib/python2.6/site-packages/vsc_base-2.4.2-py2.6.egg is being added to sys.path
  import pkg_resources
/usr/lib/python2.6/site-packages/vsc/utils/generaloption.pyc

As long as the system vsc-base is backwards compatible, that's not a big issue though (but it does suck).

python -S may be a way around this, but you can't use this when installing Python packages to force Python to pick up the vsc-install you want (it breaks setuptools).

from vsc-install.

@stdweird also, I disagree that this is not a vsc-install issue, since we're literally breaking backwards compatibility; we're not longer able to install old vsc-* packages

and that without a major version bump of vsc-install...

from vsc-install.

wait, you think that if you load a module and you get something else, this is OK? if EB can't properly install the software and module, why do you install them in the first place?
and if this is OK, it's even easier to fix, just see if you can import the python module you are trying to install, and if you can, print a message like "EB not installing package X, because, well, it doesn't matter anyway and we hope it's backwards compatible".

you can install older vsc- pacakges using an older vsc-install, if it makes you feel better we can increase the major version number, but not sure what it will change.

from vsc-install.

I'm not saying it's OK, I'd like it to b different, but I haven't found a way around it. Suggestions welcome.

from vsc-install.

Actually, we did figure this out for EB, and implemented a fix in the easyblock for EasyBuild, i.e. trash setuptools.pth in the install prefix because it results in reeling in system Python packages, cfr. easybuilders/easybuild-easyblocks#813.

It's not clear to me why this happens now with vsc-install too though...

I guess I'll have to dive into the magic that happens in shared_setup.py.

from vsc-install.

this error message is exactly what prevents the warning about
UserWarning: Module vsc was already imported from None, but /vscmnt/gent_vulpix_apps/_apps_gent/SL6/sandybridge/software/vsc-base/2.4.2/lib/python2.6/site-packages/vsc_base-2.4.2-py2.6.egg is being added to sys.path

We spent all this time on trying to get rid of it, and now you want it back? :p
I do see that you sometimes would want an old vsc-base for reproducing certain things, but then use an old vsc-install to install it, should work.

I do agree that this check breaks backwards compatibility, but the old stuff was doing weird things, that are fixed and are checked for now. So, wont fix.

from vsc-install.

I'm not saying the check shouldn't be there, it's just very annoying that this breaks installing older versions... Also, I obviously don't want the warning back.

Anyway, it doesn't seem to matter too much, since the declare_namespace stuff is effectively making it impossible to make a particular non-system installation of a vsc-* package 'win' over the system-wide installed version... So even if we could still install older versions, there's no use, since they won't be used at runtime.

I see no way in which the declare_namespace in /usr/lib/python2.6/site-packages/vsc/install/__init__.py can be escaped when using the system Python, which results in that path getting preference; the only way I can find is python -S, but then other things are broken too...

For now, this is just annoying, it shouldn't break anything, but sooner or later it will, when the system vsc-base breaks backward compatibility. Other than installing old versions of vsc-* packages that is, because vsc-install broke backward compatibility already...

W.r.t. EasyBuild, it's sort of fine, since other sites should not be affected (unless they have vsc-install installed system-wide).

from vsc-install.

well, it only breaks packages that where doing things wrong, new vsc-base won't fail this check, so even if it does something you don't like in future releases, you would still be able to install the current version, right?

from vsc-install.

ok, the issue of having setuptools push system things right in front of the path, right,
python -S will make sure you don't pick up system libraries, you will probably need to install pkt_resources and setuptools in a local path and addt his to your PYTHONPATH if you want vsc-install to keep working, but once you do this you should be isolated from your system environment. (or for easybuild, load the setuptools module)
but always running python -S in easybuild is probably something you want to do anyway?

from vsc-install.