bcgsc / rsempipeline Goto Github PK

rsempipeline is a pipeline for analyzing GEO data using RSEM. The typical analysis process is as follows:

The input to the pipeline are mainly from two resources,

• soft files for all Series (aka. GSE)
• A GSE_species_GSM.csv file which contains a list of all interested samples (aka. GSM) to be processed

There are three steps included in this pipeline:

1. Download the sra files for all GSMs from GEO website using aspc from Aspera or wget (in case when aspc fails). aspc and wget use different urls which are linked to copies of the same file.
2. sra files are converted to fastq.gz files using fastq-dump from SRA Toolkit
3. Run rsem-calculate-expression from RSEM package with all fastq.gz files for all GSMs

The pipeline is designed to run the first two steps (computationally cheap) on a localhost. Step 3 (computationally expensive) is run on a HPC cluster (e.g. genesis, westgrid cluster).

Typically, about 100 GSEs and a few thousands of GSMs are picked by our collaborators and grouped into a batch. Step 1 and 2 are done in a sub-batch-by-sub-batch fashion where all GSMs of a sub-batch are processed in parallel until finished. Each sub-batch of GSMs are selected based on their file sizes (estimated from sizes of sra and its resultant fastq.gz files) and how much disk space available on the localhost as specified in a configuration file (rp_config.yml). At the end of the second step, a submission script will be generated for each GSM, and at Step 3 a new job will be submitted to the cluster for processing the GSM using RSEM. A control mechanism has also been implemented to avoid overuse of the cluster resources such as compute nodes and disk space. The first two steps are run by the command rp-run while the generation of the submission script and job submission are handled by the command rp-transfer.

For installation and usage instructions, please refer to INSTALL.rst and USAGE.rst.

If you have found any bugs, questions, comments, please contact Zhuyi Xue ([email protected]).