Hi, kindly help us in installing the following tools required in our min-project:

• kallisto, download link : https://github.com/pachterlab/kallisto/releases/download/v0.46.2/kallisto_linux-v0.46.2.tar.gz
• salmon, download link : https://github.com/COMBINE-lab/salmon/releases/download/v1.4.0/salmon-1.4.0_linux_x86_64.tar.gz
• Trimgalore downloa link : https://github.com/FelixKrueger/TrimGalore/archive/0.6.6.tar.gz
• subread package (for R analysis) download link : https://sourceforge.net/projects/subread/files/subread-2.0.1/subread-2.0.1-Linux-x86_64.tar.gz/download

Thank you,

Here are some of the tools that the HPC users need to have set up in the HPC. If you have some time, please install them.

• GATK4 (we already have GATK3)
• Beagle

Hi @kipkurui, @espirado

My co-instructor for the workshop will need remote access to the icipe-HPC from outside icipe, can this be set up too? #12

Cheers

The packages are DECIPHER and phangorn, which are the packages required for the last step of this pipeline. @espirado, are you around to assist?

1. USEARCH; so that we are able to test our workflow using the HPC.

Download link
@kipkurui @Espira

• Deploy redcap etl server
• Write etl service script
• Test etl service

We can setup a script that creates user accounts after the user fills the form, and the account is approved. See this resource: https://www.cyberciti.biz/faq/rhel-debian-force-users-to-change-passwords/

Hi @kipkurui, @espirado

Kindly we will need these Software available for 23 training participants for the workshop, can this be set up by when, please?

The data is also available here

Thanks

Create a shareable file system from the main node to serve as an administrative data access for the other nodes

@- [ ] Server uptime and monitoring

• Server updates and kernel maintenance scripts
• Job Management configurations using slurm
• AWS cluster Management for remote job submission
• CloudFormation template for AWS HPC infrastucture creation

Tools needed to be installed in the galay instance

error while loading shared libraries: libhts.so.2: cannot open shared object file: No such file or directory
@andrew espira

Exploring various technologies used in bioinformatic research

job management using slurm

Data management practices and open research data access

Set up slurm server on the main node and client on worker nodes

Hi Andrew! We can explore using a VM machine as the master for Slurm. Here are some resources that can be useful.

1. https://www.strike3d.it/index.php/en/2018/11/06/slurm-on-centos-6/
2. https://slurm.schedmd.com/SLUG15/SlurmVM.pdf
3. https://github.com/ciemat-tic/codec/wiki/Slurm-on-Virtual-Machines
4. https://mussolblog.wordpress.com/2013/07/17/setting-up-a-testing-slurm-cluster/

Work around

• Re in stalled slurm on the clusters
• Created deployment scrip
• Resolved node failure issues

  cause of node failure

       - Firewall was blocking node communication 
  

use these resources:

1. https://docs.qiime2.org/2019.7/install/native/
2. https://hpc.ilri.cgiar.org/qiime2-software

• AWS cluster Management for remote job submission
• CloudFormation template for AWS HPC infrastucture creation
• Elk log monitoring set up
• Install modules and updates for versions
• configure hpc02 to access modules from hpc01
• Review data management aspects for data storage and backups

The latest server is Ubuntu 18.04 for the HPC. To do this, we must get the following done:

• Find a way of backing up all the data currently available in the system
• Acquire a master node server that will support the use of job management system
• Set the system up, and reinstall all the necessary software required (Explore if there is a way of transferring current users to the upgraded server).
• Set up the job management system and recreate users
• Explore a solution for the cold storage

Please assist to install Prinseq and Pear for 16S data analysis on HPC.
Prinseq: https://anaconda.org/bioconda/prinseq
Chimeraslayer : https://www.howtoinstall.me/ubuntu/18-04/chimeraslayer/
Pear: https://anaconda.org/bioconda/pear

@kipkurui

At some point, we may explore setting up, and testing moodle for use with ENBiT and our inhouse training. The instalation is not a priority for now, but using this issue to keep track of various resources we can use.

• https://docs.moodle.org/38/en/Installation_quick_guide#Basic_Requirements
• https://docs.moodle.org/38/en/Installing_Moodle
• https://www.vultr.com/docs/how-to-install-moodle-on-centos-7

Andrew help with installing RDP on the server. Its GUI based.

Install from http://cab.spbu.ru/software/spades/

Bug on cent os 7 and open access server. Firewall automatically closed all remote access to the system. bug chaser

Status

• [ FIxed]

error while loading shared libraries: libhts.so.2: cannot open shared object file: No such file or directory

• Deploy kubeflow
• Deploy deployment to spin up nodes with jupyternitebook
• test access and usage
• test package availability and user friendliness
• Automate access and deployments
• Configure monitoring

error while loading shared libraries: libhts.so.2: cannot open shared object file: No such file or directory

We are having problem running samtools, seems libhts.so.2: library is not available @Espira

For Kennedy's project, he needs to have access to the Cytoscape GUI. One solution he has found is the use of the NoMachine. We can explore how we can set this up in the HPC.

Here is the download site: https://www.nomachine.com/download/linux&id=1

• AWS cluster Management for remote job submission
• CloudFormation template for AWS HPC infrastucture creation
• Documentation of user management system on hpc
• Check on azure cloud management
• Test on redcap etl
• Work on ansible towers

We can settle for Cent Os 7 for compatibility with the existing server. @kipkurui have settled with setting the nodes as a single cluster which can allow access of both from either. A few links on doing that
Setup Two Node Red Hat Cluster without shared storage
How to configure Two Node High Availability Cluster On RHEL/CentOS 7.6
How To Configure High-Availability Cluster on CentOS 7 / RHEL 7

I will try to get a few documentations from the other admins

Globus Connect Personal enables you to share and transfer files to and from your Linux laptop or desktop computer 

Get Requirements needs and modules that support the extensive analysis that is being undertaken within the platform

1. set up roles for various metrics
2. Galaxy administration guide
3. Galaxy user guide
4. Galaxy user support and training

Guides
Galaxy training
Galaxy course

Document

• Installation guide
• Implementation strategy
• support guide

• Update blast to the latest version
• Add taxonomic database (metagenomics)

• AWS cluster Management for remote job submission
• CloudFormation template for AWS HPC infrastucture creation
• Review data management aspects for data storage and backups
• Logging management(handling logs in the system
• troubleshooting and re configuring slurm
• Documentation of user management system on hpc
• Progress presentation hpc,redcap and cloud infrastructure

• AWS cluster Management for remote job submission
• CloudFormation template for AWS HPC infrastucture creation
• Documentation of user management system on hpc
• Create data access across the clusters
• Check on azure cloud management
• Start transition of data collection tools to new developer
• Test on redcap etl
• Work on ansible towers

This library seems to be missing or not working on the server. The error is as below;
error while loading shared libraries: libssl.so.1.0.0: cannot open shared object file: No such file or directory

@- [x] Instal RStudio

• Install JupyterHub
• Setup ansible
• Deploy galxy
• create work template module for getting started with galaxy on hpc
• review user and data management policy for hpc and publish the final document
• finalize on module set up
• Mount new server
• Assign IP address,server name and configure dhcp
• Install operating system
• set up high availability cluster with two nodes
• finalize on module set up
• Dokuwiki set up
• Server Documentation
• Set Centralized authentication management
• Galaxy configuration
• Application public access review

Documentations and links for how we are preparing our cluster for running the various sequences jobs for 16s

• Containerize sample management platform
• Create infrastucture as code template to deploy to azure
• Add instrumentation into the sample management code
• create observability with new relic
• Test features and document platform

Hi Andrew.

Been testing the Jupyer HUb, and it seems when I create an environment and install packages, they don't seem to be accessible on my notebooks. Any ideas?

PS: Am I still in the sudoers list?

• AWS cluster Management for remote job submission
• CloudFormation template for AWS HPC infrastucture creation
• Use azure cloud management for dmmg
• Work on ansible towers
• symbio vector data API to pull satellite data into project
• Icipe HPC migration to AWS(review AWS whitepaper )
• Implementation of docker containers to run jupyternotebooks

For Advanced H3ABioNet training, we will need to set up Singularity. See this documentation for HPC set up that can be useful here as well. Let's make a plan for setting this up as well.

• Deploy jupyterhub to use container
• Test access and usage
• Test compute and storage
• automate user allocation of resources