Comments (4)
Hi @marymcelroy
Your question depends on the available partitions/resources in your HPC.
The more cores you assign, the faster your job will be completed.
For example, in the Zorbas HPC, I would normally use the batch partition (1 node and 20 cores).
I would say that, if I were to run e.g. a job for 96 samples and 16S rRNA using the #SBATCH specifications and the parameters I normally use, it would take more or less a day.
(You will need to run 3 separate jobs, since you have 3 genes and thus, you will have 3 different parameters files).
Μy shell script would be something like
#!/bin/bash -l
#SBATCH --partition=batch
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=20
#SBATCH --mem=40G
#SBATCH --job-name="my_pema_job"
#SBATCH --output=my_pema_job.output
#SBATCH --requeue
module purge # unloads all previous loads
module load singularity/3.7.1 #loads singularity
singularity run -B /home1/christina/the_directory_where_the_mydata_folder_and_the_parameters_are/:/mnt/analysis /home1/christina/pema_v.2.1.4.sif
module unload singularity/3.7.1 #unloads singularity
from pema.
Hi @marymcelroy.
@cpavloud is right. Just a few more comments from my side:
- a good practice is to first run only a few samples of a marker to tune your parameters and estimate the required time for your whole dataset
- for the case of COI and your ref db, you can first check if you can train the RDPClassifier in the
sandbox
, meaning once you build yoursandbox
you can try to run the commands of step 2 in there and build your new ref db under the
/tools/RDPTools/TRAIN/
path on the image. If you can do this, then you won't have to train again and again the classifier every time you run your analysis, but you will be able to set thecustom_ref_db
asNo
and give the name of your db through thename_of_custom_db
parameter.
If this note here is making things more confusing, please ignore!
In any case, if I had to guess, I would say that you wouldn't need more than 3 days time for any of your analysis no matter what parameters you ask for. A step that can really take some time if you have a great number of otus/asvs, is the getNCBITaxId
; I suggest to have this as No
at least until you have your first results.
Good luck! 💯
from pema.
Thank you both very much for the suggestions!
from pema.
@marymcelroy I now close this issue and in case you d like to give us any feedback, please feel free to open a "new discussion".
Thanks again for your interest on PEMA.
from pema.
Related Issues (20)
- add remove_singletons option in OTU pipeline HOT 4
- Error when unzipping using a checkpoing HOT 1
- custom_ref_db doesn't work. HOT 5
- Upgrade to crest4
- provide pema main data product in a 7-level taxonomy format HOT 3
- extended final table HOT 1
- PEMA Ro-Crate HOT 1
- PEMA to create a run summary file HOT 2
- MIDORI updates HOT 2
- Unexpected Singletons in Final Table despite removeSingletons set to "Yes" HOT 5
- Error PEMA ASV inference: Directory swarm does not exists HOT 5
- 16s copy number
- When Swarm is chosen for the ITS, the names of the sequences are OtuXX and not ASVXX HOT 4
- Needed updates on "PEMA's output files.md" HOT 2
- time consuming preprocessing
- Storage usage of PEMA HOT 1
- pandaseq error "Something is wrong with this ID" HOT 17
- Long recurrent taxonomic labels HOT 1
- Wrong sample names in OutputPerSample HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pema.