nrel / hpc Goto Github PK

https://nrel.github.io/HPC/Documentation/Data-and-File-Systems/File-Systems/filesystemsio/

"Lustre parallel file system: Accessiblle across all nodes. When using this file system please familiarize yourself with the [best practices] section"

[best practices] section link is non-functional; should probably link to:
https://github.com/NREL/HPC/blob/gh-pages/Documentation/Data-and-File-Systems/File-Systems/Lustre/lustrebestpractices/index.html

Both links in HPC/general/intermediate/building-mpi-applications/ appear to go nowhere

Make the announcements and blog posts searchable

There are references to Peregrine in:

HPC/general/intermediate/building-packages/

and

HPC/general/intermediate/software-environment-basics/conda-how-to.md

Use favicon from the archive branch

So far, User Ops has been advising a conda-forge install. We can either write that up, or look for a better way. also, hidden in the git-lfs docs is the charging model... might be worth mentioning.

We want to encourage Python 3, but there are cases where Py2 environments or conversions are useful though, particularly for academic offerings written in 2 that will never get ported to 3.

Maybe the user story could be "I need to use an awful Python 2 module that will never be updated to Python 3, how can I do it?" and options (1--find an alternative so you can use up-to-date auxiliary tools; 2--convert to 3 so you can use up-to-date auxiliary tools, include link to https://docs.python.org/3/library/2to3.html; 3--set up a Python 2 custom environment).

Originally posted by @chchang6 in #26 (comment)

mpirun vs srun vs mpiexec

Broken links in HPC/general/intermediate/building-mpi-applications/

add navigation links to the building packages content...

Add info on working with HDF5 files, including Lustre striping optimizations.

_includes/announcement.html seems to be limiting to 1 (good idea for the landing page) but may not be sorting as we expect. After adding more recent announcements, I still only see November.

Also, maybe we add a link to previous announcements?

pull Git content from gh-pages-archive to the current gh-pages

The docs at https://nrel.github.io/HPC/Documentation/Data-and-File-Systems/Transferring-Data/globus/ and https://www.nrel.gov/hpc/globus-file-transfer.html differ in particular with using the CLI.

Also Globus does not appear to be installed on Eagle like described in nrel.gov docs?

Add how to reattach to a FastX session to the docs.

Summarize Lustre striping benchmarks on Eagle in a blog post - @timkphd

Update the search box to say nrel.github.io so it is site specific

We have received recent community requests for JupyterLab. Ideally, we can write up a blog or module about using jupyterlab on europa, or how to setup jupyterlab using conda.

Based on discussion with @diskwarrior document or have a workshop on strategies for packing multiple jobs into a node.

Test and update Globus docs under Managing Data -> Transferring Files.

There are links in HPC/general/intermediate/software-environment-basics/conda-how-to.md that don't point to an active page. I'm not sure if the links 'conda create', 'conda env create' and 'conda clean' were meant to point to anything or if that was a mistake

References to Peregrine on HPC/general/intermediate/building-packages/

We need a logo for gh-pages site to replace the NREL branding that was taken down.

HPC/general/beginner/bash/ has a link to https://nrel.github.io/HPC/bash that doesn't exist

The description of installing Anaconda environments given in (https://nrel.github.io/HPC/languages/python/conda.html is quite thorough. My only suggestion would be to add a small subsection for cloning an already existing environment.

It can be used in case there is an existing environment somewhere on Eagle that works without issues and one wants to use it as a basis for their own environment. By cloning it, one can then add other packages as well and use it for their own projects. This is useful e.g. for cases when one wants to utilize GPUs for reinforcement learning policy training and do not want to create an environment from scratch, which can be proved to be cumbersome (CuDNN drivers, TF-gpu, etc.)

You can clone an environment as:

conda create --clone <env_name> --name <env_name_copy>

Processes and layouts have changed. The contributing guidelines should reflect the current state as well as current expectations.

https://nrel.github.io/HPC/Documentation/Systems/Swift/running/

Several places in this document have indications of merge conflicts, especially in the code blocks.
e.g.:

ml intel-oneapi-compilers
ml intel-oneapi-mpi
export I_MPI_PMI_LIBRARY=/nopt/nrel/apps/210728a/level01/gcc-9.4.0/slurm-20-11-5-1/lib/libpmi2.so
<<<<<<< HEAD
=======
export UCX_TLS=all
>>>>>>> auxsys

<<<<<<< HEAD
srun --mpi=pmi2  ./a.out -F
=======
srun   ./a.out -F
>>>>>>> auxsys

etc. There are several places with this error that need to be corrected to the right version.

Broken link in HPC/general/intermediate/Jupyterhub/

• How to structure working groups meetings
• Guidelines for reviewing pull requests/issues

Transfer this CMS content to mss.md and update with current info: https://www.nrel.gov/hpc/mass-storage-system.html

The policy page (https://www.nrel.gov/hpc/mass-storage-system-policies.html) will stay on the CMS.

• Slurm partitions table as currently the only partition is test
• Info regarding GPU nodes not currently available
• Swift File Systems
• Add info on the ssh keys + github
• Mention --cpus-per-task for numpy etc

There's a broken link in HPC/general/intermediate/software-environment-basics/README.md labeled 'detailed guide'

Update the README in the gh-pages branch to include the docker for running the mk-docs locally

Define clear process/expectations for reviewing and merging pull requests

https://www.prestashop-project.org/maintainers-guide/reviewing-pull-requests/
https://www.prestashop-project.org/maintainers-guide/summary-github-lifecycle/

Just starting the conversation to get JH docs up on website, and flesh out in repo.

First thing, have the admins seen this? https://github.com/jupyterhub/the-littlest-jupyterhub
I hadn't seen this before; since we already have an instance, this would only be interesting if it lowered management overhead dramatically or provided options more easily than the current instance. We're well within the 100-user limit.

Second, who is part of the "Jupyterhub" team now? We probably need >= 1 ACO person who can make changes to the instance, and 1 HPC Apps person in charge of usage/documentation.

pull the bash content from the archive branch

There's an internal link in HPC/applications/plexos-quick-start/ to confluence (https://tickets.hpc.nrel.gov/collab/display/CSCT/Eagle+Plexos+Examples). Can we post this somewhere so that both our internal and external users can access the content?

HPC/applications/plexos-hpc-walkthrough/ directory is all written for a workshop that was held on Peregrine. The content is pretty out of date but not a bad resource. It might not be worth updating but maybe archive it and reference it

HPC/general/beginner/how-to-transfer-files the readme needs updated from Peregrine and how to transfer files. Could use another markdown referencing something like FileZilla

Hi,

I'm following the steps (https://github.com/NREL/HPC/tree/master/workshops/Optimized_TF) for creating an environment similar to py38tf24.yml but when I reach the point to install the precompiled TensorFlow installation from a wheel, I get the following error:

ERROR: tensorflow-2.4.0-cp38-cp38-linux_x86_64.whl is not a supported wheel on this platform.

Any ideas?

Asses the layout and navigation of the GH pages site.

There's a broken link in HPC/general/beginner/intro-to-linux/ labeled 'HPC University - Introduction to Linux'

Singularity page still references Peregrine. Also additional suggested changes:

• Add GPU section
• Add a comparison of Docker / Singularity
• Add recommendation of making Docker images which can be pulled as Singularity containers for portability
• Check status of singularity.automounts = True for read/write in new version of Singularity on Eagle
• Steps to create Singularity container from Docker daemon directly
• Stub for NREL's Harbor registry.

Issue for catching inspiration or suggested designs for the github pages site:

• https://support.nesi.org.nz/hc/en-gb
• https://docs.nersc.gov/

It looks like the ML content was lost when migrating to the new layout.