generated devel module needs work about easybuild-framework HOT 10 OPEN

It has been removed from the prepare function. So this bug is solved.

But we should now create a devel-module after (or before) each step, placing this in the build directory.
This would allow for easy debugging after a build has crashed somewhere:

• change to build dir
• load the devel module for this step
• reproduce the problem
• debug the problem
• fix it
• restart build.

from easybuild-framework.

This issue was fixed by c8706a9.

Created a new issue for the step-wise devel modules, see #217

from easybuild-framework.

Loading devel modules (still?) fails:

-bash-3.2$ module load gnu/openmpi/olf/1.4.10/libpng/1.5.11 
-bash-3.2$ echo $EBDEVEL
$EBDEVELFFTW       $EBDEVELGOMPI      $EBDEVELHWLOC      $EBDEVELOPENBLAS   $EBDEVELSCALAPACK  
$EBDEVELGCC        $EBDEVELGOOLF      $EBDEVELLIBPNG     $EBDEVELOPENMPI    $EBDEVELZLIB       
-bash-3.2$ echo $EBDEVELLIBPNG 
/user/scratch/gent/vsc400/vsc40023/easybuild_REGTEST/SL6/sandybridge/software/gnu/openmpi/olf/1.4.10/libpng/1.5.11/easybuild/gnu-openmpi-olf-1.4.10-libpng-1.5.11-easybuild-devel
-bash-3.2$ module load $EBDEVELLIBPNG 
gnu-openmpi-olf-1.4.10-libpng-1.5.11-easybuild-devel(5):ERROR:105: Unable to locate a modulefile for 'gnu-openmpi-1.4.10-fftw-3.3.3-easybuild-devel'
gnu-openmpi-olf-1.4.10-libpng-1.5.11-easybuild-devel(9):ERROR:105: Unable to locate a modulefile for 'gompi-1.4.10-easybuild-devel'
gnu-openmpi-olf-1.4.10-libpng-1.5.11-easybuild-devel(13):ERROR:105: Unable to locate a modulefile for 'gnu-4.7.2-hwloc-1.6.2-easybuild-devel'
gnu-openmpi-olf-1.4.10-libpng-1.5.11-easybuild-devel(17):ERROR:105: Unable to locate a modulefile for 'goolf-1.4.10-easybuild-devel'
gnu-openmpi-olf-1.4.10-libpng-1.5.11-easybuild-devel(21):ERROR:105: Unable to locate a modulefile for 'gnu-4.7.2-openmpi-1.6.4-easybuild-devel'
gnu-openmpi-olf-1.4.10-libpng-1.5.11-easybuild-devel(25):ERROR:105: Unable to locate a modulefile for 'gnu-openmpi-1.4.10-scalapack-2.0.2-OpenBLAS-0.2.6-LAPACK-3.4.2-easybuild-devel'
gnu-openmpi-olf-1.4.10-libpng-1.5.11-easybuild-devel(29):ERROR:105: Unable to locate a modulefile for 'gcc-4.7.2-easybuild-devel'
gnu-openmpi-olf-1.4.10-libpng-1.5.11-easybuild-devel(33):ERROR:105: Unable to locate a modulefile for 'gnu-openmpi-1.4.10-openblas-0.2.6-LAPACK-3.4.2-easybuild-devel'
gnu-openmpi-olf-1.4.10-libpng-1.5.11-easybuild-devel(37):ERROR:105: Unable to locate a modulefile for 'gnu-openmpi-olf-1.4.10-zlib-1.2.7-easybuild-devel'
-bash-3.2$ echo $MODULEPATH
/user/scratch/gent/vsc400/vsc40023/easybuild_REGTEST/SL6/sandybridge/modules/all:/etc/modulefiles/vsc

from easybuild-framework.

A couple of things need to be taken care of here:

• loading a devel module should be fixed, this probably involves modifying $MODULEPATH such that devel modules of dependencies can be located
• we need to revisit the naming of the devel module, it's probably better to use the software module name and simply add /devel at the end
• loaded devel modules should appear nicely in module list, i.e. not full path; this also involves manipulating $MODULEPATH

It might be wise to simply throw all devel modules in a dedicated directory tree, e.g. modules/devel_modules, to avoid an explosion of $MODULEPATH which would probably have significant performance impact

from easybuild-framework.

• loading a devel module should be fixed, this probably involves modifying $MODULEPATH such that devel modules of dependencies can be located

This is a viable option, iff your modules/devel_modules concept is adhered to
(ie. expand once, to pick the development environment configurations)

• we need to revisit the naming of the devel module, it's probably better to use the software module name and simply add /devel at the end

Makes a lot of sense for naming schemes that do nesting (esp. in custom ModuleNameSpaces).
Note that this idea will not work on certain systems (inability to handle properly two / operators),
so it should rather be considered a nice-to-have, alternative schemes should be possible, too.

• loaded devel modules should appear nicely in module list, i.e. not full path; this also involves manipulating $MODULEPATH

hm... I had to read the above a couple of times to grok it, but now it's clear:

• In order to avoid that situation, the optimal technique should be something
  of the following type, as you said:

It might be wise to simply throw all devel modules in a dedicated directory tree, e.g. modules/devel_modules, to avoid an explosion of $MODULEPATH which would probably have significant performance impact

btw.
we already experience some delays, when people do module avail for the first time, with O(1000) modules;
the development module tree will just make that time double... any ideas about how to speed things up?
Is it the case that Lmod is more competitive as regards development environments?

from easybuild-framework.

I'm starting to wonder whether loading devel modules of the dependencies is actually required... All the devel module does is allowing to reproduce the environment in which the module was built, thus there probably is no need to load the devel modules of the dependencies (the environment in which they were built is no longer relevant?).

I can't think of an example straight away that contradicts this (since EasyBuild takes care that everything is set correctly for the dependencies, and thus those settings will be part of the single devel module we load).

from easybuild-framework.

Good question, because I was also wondering about the same:

even if you have a stack of modules which all depend on
a depend-builddepend-depend-builddepend sequence,
only the top module really matters... no?

On Tue, Oct 1, 2013 at 1:59 PM, Kenneth Hoste [email protected]:

I'm starting to wonder whether loading devel modules of the dependencies
is actually required... All the devel module does is allowing to reproduce
the environment in which the module was built, thus there probably is no
need to load the devel modules of the dependencies (the environment in
which they were built is no longer relevant?).

I can't think of an example straight away that contradicts this (since
EasyBuild takes care that everything is set correctly for the dependencies,
and thus those settings will be part of the single devel module we load).

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/109#issuecomment-25443167
.

from easybuild-framework.

Issue #203 (now closed) is a duplicate.

from easybuild-framework.

@citibob: I think you intended to mention that issue #1531 is a duplicate of #203? Seems like you misplaced your last comment?

from easybuild-framework.

Yes I think you are right.

On Sun, Jan 3, 2016 at 10:31 AM, Kenneth Hoste [email protected]
wrote:

@citibob https://github.com/citibob: I think you intended to mention
that issue #1531
#1531 is a
duplicate of #203
#203? Seems like
you misplaced your last comment?

—
Reply to this email directly or view it on GitHub
#109 (comment)
.

from easybuild-framework.