eessi / docs Goto Github PK

View Code? Open in Web Editor NEW

13.0 13.0 26.0 135.46 MB

Documentation for the European Environment for Scientific Software Installations (EESSI)

Home Page: https://eessi.github.io/docs

License: GNU General Public License v2.0

Makefile 0.38% Python 92.77% Shell 6.86%

docs's People

Contributors

Stargazers

Watchers

docs's Issues

Make https://eessi.github.io/ (webserver root) work

https://eessi.github.io/ (without docs/) now returns a 404 error page. We probably should put something (anything) there, or redirect to https://eessi.github.io/docs/ or https://www.eessi-hpc.org/?

Software layer/Build nodes documentation out of date

The documentation at the Build nodes page is out of date. Many of the installation steps have been moved to build_container.sh and some of the hard-coded paths have changed since the documentation was written.

Since new software installation is now achieved mainly through the bot, perhaps this page could be revised with up to date information for the purpose of manually debugging cases that require a more manual inspection of each step.

Document installation and configuration of Squid proxy

We should add a section in the documentation that explains how a Squid proxy can be set up. We could make use of the template in the filesystem layer repo; this still has to be updated for the new repo, though.

N.B.: it should also mention that if the cvmfs-config-default is being used by clients to connect to EESSI, the proxy should allow the Cern Stratum 1 servers in their destination ACL! Otherwise the configuration repo, and hence EESSI configuration files, cannot be accessed, as we found out during the best practices tutorial 😉

Simplify documentation

I think the general rule for the docs should be: Don't scare people into thinking this is too complex

As a reader with an above average attention span (there there people that are better than me but more that are worse) I would recommend to hide any complexity of EESSI in the documentation and basically have 3 main entry points:

Using EESSI by persona
1.1 Sysadmin (person with root): install packages and source the stuff in /etc/profiles.d
1.2. User: ask sysadmin to install packages and source yourself and/or put it in your profile
contributing to EESSI : https://www.eessi.io/docs/adding_software/opening_pr/
Want to know more (background info, policies, slurm clusters, etc ..... most things that people never read. Those also should be de-prioritized in the mkdocs-material searches so not confuse people

I find this quote explains it all:
"What if there was a way to avoid having to install a broad range of scientific software from scratch on every HPC cluster or cloud instance you use or maintain, without compromising on performance?"
The concept of easystack yaml files is probably a good idea but an entirely new concept to very infrequent users like me (just read about it today) . Is there not a way to autogenerate a yaml file in eb with default settings if i want to just submit a single easyconfig?

add "Onboarding" page to docs

We should document how to "onboard" new people to the project:

send them an invitation to the Slack channel
add them to the mailing list (via @bedroge or @zilverberg)
add them to the EESSI GitHub organization
let them open a PR to add themselves to the "Partners" page
quick introduction via email + next EESSI conf call

Images are loading slow?

Images are loading a bit slow on my machine/connection, maybe we can improve this by optimizing the size or the resolution?

document available resources

AWS (@terjekv, @zilverberg, @bedroge, @boegel, @kniec)
Fenix (@zilverberg)
OSU OSL for POWER9 (@bedroge)

add documentation on build nodes

see EESSI/software-layer#40 as starting point

next steps:

export EESSI_PILOT_VERSION=2020.12
/cvmfs/pilot.eessi-hpc.org/${EESSI_PILOT_VERSION}/compat/linux/$(uname -m)/startprefix
./EESSI-pilot-install-software.sh

Also need to document:

$EESSI_SOFTWARE_SUBDIR_OVERRIDE to override subdirectory in which software is installed
--generic option for EESSI-pilot-install-software.sh script

Describe client minimum requirements

I'd like to see a chapter in the docs describing what a client needs at minimum to start making use of EESSI repository, both in a free text form and in some container definition form such as Dockerfile. Aim is to inform infrastructure architects about what they need to provide to enable adoption of EESSI.

Document setup

Add documentation on setup of an EESSI cluster.
Add example automated setup scripts that can be used on generic clouds.

CUDA SDK in host_injections cannot be installed by root

Running the install_cuda_host_injections.sh as root doesn't work, since EasyBuild doesn't allow it:

{EESSI 2023.06}                                                                                                                                                                                                        
15:03 login-gpu1.liza.surf.nl:/tmp                                                                                                                                                                                     
root# /cvmfs/software.eessi.io/versions/${EESSI_VERSION}/scripts/gpu_support/nvidia/install_cuda_host_injections.sh -c 12.3.0 --accept-cuda-eula 
Attempting to load an EasyBuild module to do actual install                                                                                                                                                            
Traceback (most recent call last):                                                                         
  File "<frozen runpy>", line 198, in _run_module_as_main                                                 
  File "<frozen runpy>", line 88, in _run_code                                                             
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/software/EasyBuild/4.8.2/lib/python3.11/site-packages/easybuild/main.py", line 755, in <module>
    init_session_state, eb_go, cfg_settings = prepare_main()                                                                                                                                                           
                                              ^^^^^^^^^^^^^^                                                                                                                                                           
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/software/EasyBuild/4.8.2/lib/python3.11/site-packages/easybuild/main.py", line 749, in prepare_main
    eb_go, cfg_settings = set_up_configuration(args=args, logfile=logfile, testing=testing)                                                                                                                            
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                            
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/software/EasyBuild/4.8.2/lib/python3.11/site-packages/easybuild/tools/options.py", line 1566, in set_up_configuration
    check_root_usage(allow_use_as_root=options.allow_use_as_root_and_accept_consequences)                                                                                                                              
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/software/EasyBuild/4.8.2/lib/python3.11/site-packages/easybuild/tools/options.py", line 1518, in check_root_usage
    raise EasyBuildError("You seem to be running EasyBuild with root privileges which is not wise, "                                                                                                                   
easybuild.tools.build_log.EasyBuildError: "You seem to be running EasyBuild with root privileges which is not wise, so let's end this here."                                                                           
Traceback (most recent call last):                                                                                                                                                                                     
  File "<frozen runpy>", line 198, in _run_module_as_main                                                                                                                                                              
  File "<frozen runpy>", line 88, in _run_code                                                             
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/software/EasyBuild/4.8.2/lib/python3.11/site-packages/easybuild/main.py", line 755, in <module>
    init_session_state, eb_go, cfg_settings = prepare_main()                                               
                                              ^^^^^^^^^^^^^^                                               
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/software/EasyBuild/4.8.2/lib/python3.11/site-packages/easybuild/main.py", line 749, in prepare_main
    eb_go, cfg_settings = set_up_configuration(args=args, logfile=logfile, testing=testing)                
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/software/EasyBuild/4.8.2/lib/python3.11/site-packages/easybuild/tools/options.py", line 1566, in set_up_configuration
    check_root_usage(allow_use_as_root=options.allow_use_as_root_and_accept_consequences)                  
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/software/EasyBuild/4.8.2/lib/python3.11/site-packages/easybuild/tools/options.py", line 1518, in check_root_usage
    raise EasyBuildError("You seem to be running EasyBuild with root privileges which is not wise, "                                                                                                                   
easybuild.tools.build_log.EasyBuildError: "You seem to be running EasyBuild with root privileges which is not wise, so let's end this here."
ERROR: The easyconfig CUDA-12.3.0.eb was not found in EasyBuild version:                                   
  This is EasyBuild 4.8.2 (framework: 4.8.2, easyblocks: 4.8.2) on host login-gpu1.liza.surf.nl.           
You either need to give a different version of CUDA to install _or_                                        
use a different version of EasyBuild for the installation.                                                 
                                                                                                           
The versions of available with the current eb command are:

Since this script is actually not unlikely to be run as root, maybe we should add the possibility of passing an --as-root argument, that then sets the --allow-use-as-root-and-accept-consequences argument to EasyBuild?

Improvements for build nodes documentation

When using something else than /tmp for $EESSI_TMPDIR, you may have to bind mount that location by passing -B $EESSI_TMPDIR to Singularity. We can probably just always use this flag in the instructions, even when /tmp is being used.
Maybe have a TLDR section that allows you to easily copy all commands in one go.
Add an extra warning about the working directory from which the tarball gets created; this really has to be a subdir of the repo, as illustrated in the example.
Add a section on how this can be run as a (non-interactive) job. (using startprefix <<< buildscript.sh)
Build nodes should have internet access
How to set the number of cores to be used for compiling
Easybuild should use a temporary dir in $EESSI_TMPDIR (which can be set to something else than /tmp), or we should bind mount a subdir of $EESSI_TMPDIR as /tmp in the container
Check available disk space (~30GB?)
Make the tarball (especially when you install some additional packages, but it can be done in any case) in ${EESSI_TMPDIR}/overlay-upper/${EESSI_PILOT_VERSION}/software/linux

eessi / docs Goto Github PK

docs's People

Contributors

Stargazers

Watchers

Forkers

docs's Issues

Recommend Projects

Recommend Topics

Recommend Org