easybuilders / easybuild-framework Goto Github PK

Hi there,

just fetched most recent version and got the following; (including for real builds)
I guess it is some "work-in-progress-snapshot"!

sw@gaia-11:~$ ./easybuild/eb --dep-graph=depgraph.dot --robot ~/easybuild/easybuild/easyconfigs easybuild/easybuild/easyconfigs/h/HPL/HPL-2.0-goalf-1.1.0-no-OFED.eb
== resolving dependencies ...
Traceback (most recent call last):
  File "/usr/lib/python2.6/runpy.py", line 122, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.6/runpy.py", line 34, in _run_code
    exec code in run_globals
  File "/mnt/nfs/users/homedirs/sw/easybuild/easybuild/build.py", line 863, in <module>
    main()
  File "/mnt/nfs/users/homedirs/sw/easybuild/easybuild/build.py", line 272, in main
    orderedSpecs = resolveDependencies(packages, options.robot, log, force=force)
  File "/mnt/nfs/users/homedirs/sw/easybuild/easybuild/build.py", line 492, in resolveDependencies
    processedSpecs = processEasyconfig(path, log, validate=(not force))
  File "/mnt/nfs/users/homedirs/sw/easybuild/easybuild/build.py", line 394, in processEasyconfig
    eb = EasyConfig(spec, validate=validate)
  File "/mnt/nfs/users/homedirs/sw/easybuild/easybuild/framework/easyconfig.py", line 152, in __init__
    self.parse(path)
  File "/mnt/nfs/users/homedirs/sw/easybuild/easybuild/framework/easyconfig.py", line 167, in parse
    execfile(path, global_vars, local_vars)
  File "/home/clusterusers/sw/easybuild/easybuild/easyconfigs/g/goalf/goalf-1.1.0-no-OFED.eb", line 38, in <module>
    ('ScaLAPACK','1.8.0', '-%s-%s-%s-BLACS-%s' % (comp_mpi_tk, blas, lapack, blacsver))
NameError: name 'lapack' is not defined
sw@gaia-11:~$ 

(old internal ticket 318)

Currently, EasyBuild relies on having the user provide an easyconfig file which specifies all the build parameters: software name and version, toolkit, patch files, etc.

It might be worthwhile to implement a way to run EasyBuild without having to supply an easyconfig, i.e. by just specifying the name of the software to build, and optionally a version, toolkit (version), versionprefix, etc.

EasyBuild can then try and find either a matching easyconfig file given the desired specs, pick some reasonable values for parameters that were not supplied (e.g. build the latest known software version if no version was specified), or generate an easyconfig based on the available ones (e.g., just change the version/toolkit when reasonable, and attempt to build).

This way, we could simply supply template easyconfig files in the easybuild/easyconfigs directory, and add more specific easyconfigs if needed (e.g. if certain patch files are required to build with an ictce toolkit).

Implementing this would also allow to supply an easyconfig and override some of the parameters set in that file, e.g. software version, toolkit, etc.

The mtl download url in the example .eb's is broken.

http://www.simunova.com/downloads/mtl4/MTL-all-4.0.8878-Linux.tar.gz returns a 404
This is because MTL4 updates their tarbals every thursday to the latest svn revision.
If you want a fixed version you should use their public svn
svn checkout https://simunova.zih.tu-dresden.de/svn/mtl4/trunk mtl4

(old internal ticket 74)

After (correctly) installing a software package, we should make the installation directory read-only, to avoid accidentally overwriting (part) of the installation.

When doing a forced reinstallation (with or without skip), we should make sure to make it writeable again.

Because we now allow arbitrary code to be executed in easyconfig (.eb) files, that's a major security issue.

Easyconfigs files can easily be passed between users of EasyBuild, and they can execute totally arbitrary code (e.g. rm -rf * in the user's home directory).

We need to find a way to remedy that.

One suggestions is to limit what can be done in an easyconfig file (suggested by @nudded):

global_vars = {
               "shared_lib_ext": get_shared_lib_ext(),
               "__builtins__": {}
              }

Hi,

we should decide how best to manage the following; I'd prefer the "module" style resolution,
but it is worthy to understand how to handle the osresolution business in such case.

grep zlib ./easybuild/easyconfigs/p/Python/Python-3.2.3-goalf-1.1.0-no-OFED.eb

osdependencies=['zlib-devel']

fgeorgatos@gaia-1:~/easybuild$ dpkg -l 'zlib*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                   Version                                Description
+++-======================================-======================================-============================================================================================
un  zlib-bin                               <none>                                 (no description available)
un  zlib-dev                               <none>                                 (no description available)
un  zlib-gst                               <none>                                 (no description available)
un  zlib1                                  <none>                                 (no description available)
un  zlib1-dev                              <none>                                 (no description available)
ii  zlib1g                                 1:1.2.3.4.dfsg-3                       compression library - runtime
un  zlib1g-dbg                             <none>                                 (no description available)
ii  zlib1g-dev                             1:1.2.3.4.dfsg-3                       compression library - development
un  zlibc                                  <none>                                 (no description available)

(old internal ticket 224)

Force installing selected (R, Python, ...) extensions should be possible somehow, for example when the easyblock for installing some extension has had some changes, and only an extension needs to be reinstalled (as opposed to the entire module and all extensions).

(old internal ticket 16, 110)

We should try and support non-GNU Linux systems, for example OS X.

Known issues:

• no /proc/cpuinfo
• commands liketar have different options
• locale issues
  • when running EasyBuild in a session that orginates from on OS X system, you might run into problems like:

Catastrophic error: could not set locale "" to allow processing of multibyte characters

This can be fixed by setting the following environment variables on the OS X system:

export LANG=C; export LC_ALL=en_US

Some of the functions (mainly the ones being called in the build function of build.py) could be named better.

E.g., make_install should be changed to install, make to build, etc.

@fgeorgatos will study if there's a consensus on a naming scheme by looking at porting systems (e.g., MacPorts).

wishlist by Fotis

Hi there,

I've got some trouble while trying to build this branch:
https://github.com/fgeorgatos/easybuild/tree/86_encoding_scheme

the errors I get are visible here: http://pastebin.com/SLnY3rSw

I recommend you rather ignore them, just try the branch and tell if the issue is on my side ;-)

thanks,
Fotis

(old internal ticket 215)

A contributor should preferably clone the main EasyBuild GitHub repository, implement easyblocks in there and then contribute them back (see also [[Contributing back]]).

If for some reason easyblocks can not be contributed back, they can also be kept in a private easyblocks repository.

Basically, you just need to make sure these private repositories are in the PYTHONPATH, and everything should work out fine.

(old internal ticket 223)

We should support specifying the setting of environment variables in module files, in the easyconfig.

For example:

moduleVars = {'EXTRA_VARIABLE':'some value'}

(old internal ticket 328)

For example:

EasyBuildError: "EasyBuild crashed with an error (at easybuild/easybuild/tools/filetools.py:146 in extractCmd): Unknown
file type from file /home/clusterusers/claczny/easybuild_sources/b/bowtie2/bowtie2-2.0.0-beta7-linux-x86_64.zip/download

This is not an EasyBuild crash, but should be an error to inform the user that something is wrong with his easyconfig file.

Tell him the supported filetypes. And to use a tuple to describe the download url if there are suffixes behind the name-version.tgz part.

(old internal ticket 321)

The CP2K instructions to build with intel are on http://software.intel.com/en-us/articles/build-cp2k-using-intel-fortran-compiler-professional-edition/.

These are a bit different compared to what EasyBuild has implemented now in v0.8 .

We need to:

• distinguish between v2.1 and v2.2 branches
• fewer exceptions for .F files
  • latest easyblock had 4 exceptions, only 1 or 2 seem really needed
• -lfftw3xf_intel needs to be added to the libs in order to actually use the FFTMKL

Once this is done, we should contact the Intel guy who updated the page to promote EasyBuild, see
http://software.intel.com/en-us/profile/71327/.

(old internal ticket 235)

replaces ARPACK, has mpi version too

very similar to current ARPACK, e.g. for gimkl toolkit just configure/make/make install:

export F77=gfortran
export MPIF77=mif90
export CC=mpicc
 ./configure --prefix=/tmp/arpack_ng --with-pic --with-blas="-L$SOFTROOTIMKL/lib/em64t -lmkl_gf_lp64 -lmkl_sequential -lmkl_core -lpthread" --with-lapack="-L$SOFTROOTIMKL/lib/em64t -lmkl_gf_lp64 -lmkl_sequential -lmkl_core -lpthread" --enable-mpi
make
make install

For example:

/home/kehoste/.local/easybuild/software/zlib/1.2.7-goalf-1.1.0-no-OFED/lib/libz.so.1: no version information available

also see http://superuser.com/questions/305055/how-to-diagnosis-and-resolve-usr-lib64-libz-so-1-no-version-information-avail

This is caused by programs/libraries that both use zlib and other (system) libraries at the same time.

The linker ld notices that another library was linked against another version of zlib, and spit out this error message.

Since the error message is printed every time libz.so is loaded, this is annoying.
Also, sometimes it breaks things, for example during the PETSc build (it thinks the ar archiver doesn't work properly because the "no version information available" error message is being printed when ar is tested).

(old internal ticket 9)

(note: keep this overview up-to-date by editing this issue, and mention new promo-actions as a reply)

Promote EasyBuild on suitable venues:

conferences/magazines:

• HEPIX spring workshop 2012 (@JensTimmerman)
• Supercomputing 2012 (@boegel, @JensTimmerman) (see notes)
  • paper at PyHPC-2012 (@ SC'12) (available here)
  • BoF session @ SC12 (not accepted) (but we did join in on Python for HPC and Small Clusters BoFs during SC12)
  • walk exhibit floor with EasyBuild business cards, talk to sysadmins of big sites (see notes)
• try and get an article in HPC-admin magazine, see e.g. article on modules (talked to editor @ SC'12, he would mention EasyBuild to the writers)

websites:

• EasyBuild webpage
• Wikipedia : add http://en.wikipedia.org/wiki/EasyBuild
• http://openscience.org/
• Intel documentation on building HPC software (e.g. see here), contact people setting this up (e.g. http://software.intel.com/en-us/blackbelt)
• HPC Advisory Network of Expertise
• Scipy once we support it (see easybuilders/easybuild-easyconfigs#35)
• http://www.rescale.com/

software-specific

• DOLFIN:
• stackoverflow question on building fenics/dolfin
• let the guy who wrote dorsal know about EasyBuild

other

• order EasyBuild business cards
• order EasyBuild stickers
• promote EasyBuild in bioinformatics community (@fgeorgatos)
• begrid

The Pasha example easyconfig file that uses that goalf toolkit now depends on a commercial version of Intel TBB.

We should change that, by adding support for the open-source TBB version, and letting Pasha-goalf depend on that.

During testing for #103, I noticed that dependencies (e.g. zlib for Python) are not loaded when only packages are being built, and the Python build/installation itself it being skipped by using the -k command line option.

When building extra packages with Python 3.2.3, this results into problems like:

  File "/home/kehoste/.local/easybuild/software/Python/3.2.3-goalf-1.1.0-no-OFED/lib/python3.2/zipfile.py", line 15, in <module>
    import binascii
ImportError: libz.so: cannot open shared object file: No such file or directory

command line used:

eb Python-3.2.3-goalf-1.1.0-no-OFED.eb -ldkf

(used to be internal ticket 8)

We should generate documentation for EasyBuild based on the class/function docstrings using Sphinx, and make it part of the release (e.g., in a doc folder).

example at http://studwww.ugent.be/~pdbaets/easybuild/

Dedicated classes for installing Python packages in an existing Python installation are currently being put in python.py. Examples are Nose, Numpy, Scipy, FortranPythonPackage, DefaultPythonPackage.

We should move these to dedicated modules, and preferably in such a way that they can both be installed as separate modules and as part of an existing Python module.

Some packages have a .sig file, e.g. gcc
ftp://ftp.gnu.org/gnu/gcc/gcc-4.4.4/

we should check the signature, and compare the key with a trusted one.

(old internal tickets 323, 344, 345)

We should set up a 'project' on a Jenkins server to run the EasyBuild regression test, which will significantly help in assuring robustness of EasyBuild.

The unit test framework (`python -m easybuid.test.suite) should be run very regularly, e.g. after every commit to the develop branch, since this is very light-weight.

We should also build all the supported software packages from scratch (i.e. with an empty MODULEPATH, and using the --robot), for example every week.

For every release, a full regression tests also needs to be run an tracked by the Jenkins setup.

The --dump-classes command line option yields no output at all:

[kehoste@ca60c171 easybuild1]$ ./eb --dump-classes
[kehoste@ca60c171 easybuild1]$ 

It's broken both on v0.8 and on the current develop branch.

The wiki page on compiler toolchains (https://github.com/hpcugent/easybuild/wiki/Compiler-toolchains) needs to be extended, with detailed descriptions of what a compiler toolkit is, why it should be used. It should be made clear that EasyBuild doesn't care how the compiler toolkit is named; it only cares which modules are (direct) depdendencies.

It should also mention and explain goalf (and ictce?) as examples, with figures (e.g. dependency graph for goalf), as examples.

Support Linux From Scratch (LFS)

see http://www.linuxfromscratch.org/

lmod is an alternative to environment modules (which has it's problems, e.g. the modulecmd segfault issue, or the way in which dependencies are handled)

We should add support for lmod to EasyBuild, without destroying the support for environment modules.

see http://www.tacc.utexas.edu/tacc-projects/mclay/lmod

The parallel builder downloads sources before it submits jobs to perform the build (this helps avoid multiple jobs downloading the same file and creating a mess).

However, it currently doesn't check whether those downloads succeeded, and just submits the job anyway.

This should be fixed, the parallel builder should make sure all the required sources and patches are found before submitting the jobs.

This can be done by making Application.prepare_build return a boolean indicating if it's OK to proceed, and checking that return value in parallelbuild.prepare_package and parallelbuild.build_packages_in_parallel.

We need an encoding scheme to handle problems with names of software packages that would yield identifical module and/or class names via the current scheme (lowercase module names, camel-case class names).

Clashes would occur for e.g. 'R' / 'r', 'LAPACK' / 'LaPack' / 'lapack'.

Problems would also occur for e.g. '7zip' (class names can't start with a number), '_' or '-' or 'c++' or '|' .

We should split up the Application class into EasyBlock and a more trimmed down version of Application, i.e., everything that is a default implementation (e.g. configure, make, make_install, ...).

Hi,

the point of this is to ensure that there is no strict need to rewrite an .eb file when the sysadmin knows that the
osdependencies are somehow available. This may be the case with items like tcsh, zlib, devel-libs etc.

This option is practically a must for people trying easybuild in a non-redhat environment!

Fotis

Currently, generic classes where easyblocks can extend from are mixed with software-specific easyblocks in the easyblocks module.

We should (consider to) change this, and move the generic support, i.e. stuff that's not tied to a particular software package, to the framework module instead.

One exception is PythonPackage because this are tied to Python.

CMake should be removed (?), and easyblocks extending this class (e.g. CMakePythonPackage) should just call a cmake_configure function that's is hosted somewhere in framework.

(old internal ticket)

EasyBuild is heavily used already on RedHat based systems (Scientific Linux 5/6, Fedora), and also on Debian systems.

It's probably worthwhile to test EasyBuild on different Linux distros, see what works and what not, and how to fix the issues that occur.

E.g., :

• openSUSE
• Arch
• Ubuntu
• SuSe
• ...

(old internal ticket 317)

We need to be able to also specify alternative ways of downloading sources, e.g. checking out from a CVS/SVN/git/hg repository.

This is required for e.g. MTL4 (see issue #111), CP2K, ...

(old internal ticket 51)

When we forcibly reinstall a software package using EasyBuild using --force, we probably have a good reason for that.

We should remove the module first, and then the installation directory, before continuing with the installation. That way, nobody can load the module while the software is being reinstalled.

One exception is when additional packages are being installed: then we combine --force with -k to skip the existing installation and go straight to the (non-installed) packages.

(old internal ticket 121)

The Cloudera Hadoop set of tools should be (fully) supported in EasyBuild.

(support for installing Hadoop, Flume and Hive is already implemented in the old non-public version of EasyBuild, and needs to be ported)

see https://ccp.cloudera.com/display/SUPPORT/CDH3+Downloadable+Tarballs

Since a whle, the good people of CP2K project started uisng version numbers.

They release two branches (2.1 and 2.2) and one trunk (which is 2.3).

You can fetch the tarballs from :

• 2.1: ftp://ftp.berlios.de/pub/cp2k/cp2k-2_1-branch.tar.gz
• 2.2: ftp://ftp.berlios.de/pub/cp2k/cp2k-2_2-branch.tar.gz
• latest trunk: ftp://ftp.berlios.de/pub/cp2k/cp2k.tar.gz

(links taken from intel page on how to build cp2k with Intel
http://software.intel.com/en-us/articles/build-cp2k-using-intel-fortran-compiler-professional-edition/)

The version can be extracted after unpacking with grep 'version_nr[ ]*=' cp2k/src/cp2k_info.F, e.g.:

/tmp/cp2k $ tar xzf cp2k-2_1-branch.tar.gz 
/tmp/cp2k $ grep 'version_nr[ ]*='  cp2k/src/cp2k_info.F
  CHARACTER(LEN=*), PARAMETER :: version_nr      = "2.1.820"
/tmp/cp2k $ rm -Rf cp2k

We should start using this version convention as of now.

(old internal ticket 228)

We should try and get rid of exec and execfile statements that (possibly) execute arbitrary code.

They're used in several places, e.g., :

• easybuild/framework/application.py: exec("from %s import %s" % (allclassmodule, name))
• easybuild/framework/application.py: exec("from %s import %s" % (allclassmodule, defaultClass))
• easybuild/framework/application.py: exec("p=%s(self,pkg,pkginstalldeps)" % defaultClass)
• easybuild/framework/easyconfig.py: execfile(path, global_vars, local_vars)
• easybuild/tools/class_dumper.py: exec("from %s import file as moduleRoot" % root)
• easybuild/tools/config.py: execfile(filename, {}, fileVariables)
• easybuild/tools/modules.py: exec stdout

We should try and get rid of these (related to #129).

Hi guys,

please pull from https://github.com/fgeorgatos/easybuild.wiki.git

tia,
Fotis

(old internal ticket 231)

We should try and make sure EasyBuild runs (correctly) with Python v3.x.

Supporting Python 2.4 -> 3.2 can be done in a single code base.

For example handling the changed named Exception syntax can be made to work in both 2.4 and 3.1 by using

try:
  #something
except SomeException:
  ex = sys.exc_info()
  #handle exception

We should support deinstalling a module built with EasyBuild.

It should result in removing the module and install directory, but only if the specified module is not used as a dependency somewhere else (I guess this also means we should track reverse dependencies between module, which we don't right now).

The wiki page on [[Code style]] should explain our code style policy, i.e. PEP008 based with some exceptions (e.g. class names for easyblocks, cfg [[Encode class names]]).

It should make clear that the intention of the code style is too keep the management of the EasyBuild code base, with contributors from multiple sites, maintainable.

The wiki page [[Development guide]] needs a thorough update after all the changes made in v0.9 (cfg. #99).

It also needs to be extended, outlining some guidelines to e.g. implement a new easyblock (+ example easyconfig), or make adjustments to the EasyBuild framework or tools.

The parallel builder should be aware of builds that are already submitted as jobs, so that it doesn't submit the same build twice when builds multiple software packages with dependencies are being submitted one after the other.

(old internal ticket 319)

We should pay more attention to floating point precision model when building software.

For this, we should provide a config parameter fpacc, and allow it to be set to values like strict, precise, normal, loose, etc.

With the Intel compilers, that would correspond to something like:

• strict: -fp-relaxed -fp-speculation=strict -fp-model strict
• precise: -fp-model precise ?
• normal: -ftz -fp-relaxed -fp-speculation=safe -fp-model source
• loose: -fp-model fast=1 ?
• veryloose: -fp-model fast=2 ?

What this corresponds to for GCC (and other compilers) needs to be looked into.

This is relevant for software like Gaussian, CP2K and WRF, among others.

Not having payed attention to this before may explain why WRF builds (among others) with the Intel compilers v12 are not working correctly (i.e. yield wrong results).

Also see http://software.intel.com/file/39386

Generating the devel module is already started in prepare, so before configure and friends (why is it already created there?).

For easyblocks that redefine make_module_req_guess, this could be a problem, because in various occasions this function relies on self.x variables that were set during the build (e.g. for SLEPc).

So, the devel module may contain incorrect values because of that.

Also, the devel module also loads the devel module of all dependencies, which seems to be causing issues (e.g. try loading the current devel module for DOLFIN).

For every release of EasyBuild, we should update the list of supported software (List of supported software packages).

Ideally, this should be done by simply running a script that spits out the new wiki page.

(old internal ticket 132)

Support for installing FLUENT with EasyBuild should be added.

This is already partially available in the old non-public version of EasyBuild, and thus needs to be ported.
Also, these remarks need to be resolved:

• patch to enable PSM, which results in pinning (whatever FLUENT may report in the logs):

> -bash-3.2$ diff -u mpirun.fl mpirun.fl.orig
> --- mpirun.fl    2012-02-26 16:59:27.971645390 +0100
> +++ mpirun.fl.orig    2010-10-01 22:42:19.000000000 +0200
> @@ -624,7 +624,6 @@
>
>      # (set specific flags)
>
> -    my_protocol=psm
>      case $my_protocol in
>      default)
>          my_protocol_flags=
> -bash-3.2$ pwd
> /apps/gent/gulpin/magnycours/software/FLUENT/13.0/v130/fluent/fluent13.0.0/multiport/mpi_wrapper/bin

• default HP MPI needs to be changed:

> -bash-3.2$ pwd
> /apps/gent/gulpin/magnycours/software/FLUENT/13.0/v130/fluent/fluent13.0.0/multiport/mpi/lnamd64
> -bash-3.2$ ll
> total 5
> lrwxrwxrwx 1 vsc40003 vsc40003  11 Feb 26 16:25 hp -> hp-2.03.01/
> drwxrwxr-x 7 vsc40003 vsc40003 512 Oct 22  2010 hp-2.03.01
> drwxrwxr-x 6 vsc40003 vsc40003 512 Oct 22  2010 hp.old
> drwxrwxr-x 4 vsc40003 vsc40003 512 Oct 22  2010 intel
> drwxrwxr-x 5 vsc40003 vsc40003 512 Oct 22  2010 openmpi

So, basically:

mv hp hp.old
ln -s hp-2.03.01 hp

(old internal ticket 97)

Building Trinity on a Scientific Linux 6 system (Intel Westmere architecture) fails with the ismkl toolkit (Intel compilers, ScaleMP MPI and Intel MKL).

To be more specific: the build of the RSEM-mod plugin fails with ismkl/3.3.1.