Comments (12)
Hi @Sanrrone
That is seems to be a problem with your singularity setup. Can you try with Docker instead?
Cheers
from emg-viral-pipeline.
Hi!,
unfortunately I am under a HPC so I cannot use Docker.
from emg-viral-pipeline.
Hi @Sanrrone!
Ah, but when you are on a HPC please use the available job scheduler, e.g. if it is SLURM:
-profile slurm,singularity
I think you used local
which will run all processes on the current node you are (probably the login node, where no calculations should run).
Besides, on some HPCs there are latency issues. If this still happens, please let us know and we can maybe give some further guidance.
from emg-viral-pipeline.
Hi,
I'm using local but inside a slurm job just to test it.
if I use the slurm profile I got:
sbatch: error: AssocMaxSubmitJobLimit
which is a missconfiguration in the profile. How can I edit the slurm profile?
from emg-viral-pipeline.
So you submit the whole pipeline run command as a SLURM job? Hm, should also work I think.
To edit the Nextflow SLURM profile, you could clone the GitHub repository and edit the nextflow.config
file (there is a SLURM part, https://github.com/EBI-Metagenomics/emg-viral-pipeline/blob/master/nextflow.config#L99). Then, run the pipeline via specifying the nextflow run virify.nf ...
There is also a way to inject an additional configuration via the -c
flag that can overwrite pipeline defaults. But maybe it's easier to first try the other way.
from emg-viral-pipeline.
Hi,
I used the slurm profile with the local virify.nf with no success, still the process doesn't finish.
N E X T F L O W ~ version 23.04.1
Launching `/scratch/project_2007362/software/emg-viral-pipeline/virify.nf` [nice_planck] DSL2 - revision: 20ab8a0de7
Profile: slurm,singularity
Current User: valensan
Nextflow-version: 23.04.1
Starting time: 15-04-2023 06:51 UTC
Workdir location:
/scratch/project_2007362/sandro/HeP_samples/6_tax_profile/phage/HeP-1003_tmp
Databases location:
nextflow-autodownload-databases
Dev ViPhOG database: v3
Dev Meta database: v2
Only run annotation: false
[- ] process > preprocess:rename -
[- ] process > preprocess:length_filtering -
[- ] process > detect:virsorter -
[- ] process > detect:virfinder -
[- ] process > detect:pprmeta -
[- ] process > detect:parse -
[- ] process > postprocess:restore -
[- ] process > annotate:prodigal -
[- ] process > annotate:hmmscan_viphogs -
[- ] process > annotate:hmm_postprocessing -
[- ] process > annotate:ratio_evalue -
[- ] process > annotate:annotation -
[- ] process > annotate:plot_contig_map -
[- ] process > preprocess:rename -
[- ] process > preprocess:length_filtering -
[- ] process > detect:virsorter -
[- ] process > detect:virfinder -
[- ] process > detect:pprmeta -
[- ] process > detect:parse -
[- ] process > postprocess:restore -
[- ] process > annotate:prodigal -
[- ] process > annotate:hmmscan_viphogs -
[- ] process > annotate:hmm_postprocessing -
[- ] process > annotate:ratio_evalue -
[- ] process > annotate:annotation -
[- ] process > annotate:plot_contig_map -
[- ] process > annotate:assign -
[- ] process > annotate:checkV -
[- ] process > annotate:write_gff -
[- ] process > plot:generate_krona_table -
[- ] process > plot:krona -
[- ] process > plot:generate_sankey_table -
[- ] process > plot:sankey -
executor > slurm (1)
[df/c2eed8] process > preprocess:rename (1) [ 0%] 0 of 1
[- ] process > preprocess:length_filtering -
[- ] process > detect:virsorter -
[- ] process > detect:virfinder -
[- ] process > detect:pprmeta -
[- ] process > detect:parse -
[- ] process > postprocess:restore -
[- ] process > annotate:prodigal -
[- ] process > annotate:hmmscan_viphogs -
[- ] process > annotate:hmm_postprocessing -
[- ] process > annotate:ratio_evalue -
[- ] process > annotate:annotation -
[- ] process > annotate:plot_contig_map -
[- ] process > annotate:assign -
[- ] process > annotate:checkV -
[- ] process > annotate:write_gff -
[- ] process > plot:generate_krona_table -
[- ] process > plot:krona -
[- ] process > plot:generate_sankey_table -
[- ] process > plot:sankey -
executor > slurm (1)
[df/c2eed8] process > preprocess:rename (1) [ 0%] 0 of 1
[- ] process > preprocess:length_filtering -
[- ] process > detect:virsorter -
[- ] process > detect:virfinder -
[- ] process > detect:pprmeta -
[- ] process > detect:parse -
[- ] process > postprocess:restore -
[- ] process > annotate:prodigal -
[- ] process > annotate:hmmscan_viphogs -
[- ] process > annotate:hmm_postprocessing -
[- ] process > annotate:ratio_evalue -
[- ] process > annotate:annotation -
[- ] process > annotate:plot_contig_map -
[- ] process > annotate:assign -
[- ] process > annotate:checkV -
[- ] process > annotate:write_gff -
[- ] process > plot:generate_krona_table -
[- ] process > plot:krona -
[- ] process > plot:generate_sankey_table -
[- ] process > plot:sankey -
slurmstepd: error: *** JOB 17780187 ON r18c08 CANCELLED AT 2023-07-11T15:31:30 DUE TO TIME LIMIT ***
the strange thing is I have no log about it so I cannot track the issue at all.
from emg-viral-pipeline.
Ok, this is indeed strange.
Can you run other Nextflow pipelines (e.g. try smt from https://nf-co.re/)? I want to figure out if this is a specific problem with VIRify or a general problem with your architecture and Nextflow installation.
from emg-viral-pipeline.
I tried to run viroprofiler, and yes, Also with an error but this time related with singularity:
-[ViroProfiler] Pipeline completed with errors-
[- ] process > VIROPROFILER:SETUP:DB_CHECKV -
[- ] process > VIROPROFILER:SETUP:DB_VIBRANT -
[- ] process > VIROPROFILER:SETUP:DB_VIRSO... -
[- ] process > VIROPROFILER:SETUP:DB_DRAM -
[- ] process > VIROPROFILER:SETUP:DB_IPHOP -
[- ] process > VIROPROFILER:SETUP:DB_VREFSEQ -
Pulling Singularity image docker://denglab/viroprofiler-virsorter2:v0.2.5 [cache /scratch/project_2007362/software/singularity/cache/denglab-viroprofiler-virsorter2-v0.2.5.img]
Pulling Singularity image docker://denglab/viroprofiler-geneannot:v0.2 [cache /scratch/project_2007362/software/singularity/cache/denglab-viroprofiler-geneannot-v0.2.img]
Pulling Singularity image docker://denglab/viroprofiler-vibrant:v0.2 [cache /scratch/project_2007362/software/singularity/cache/denglab-viroprofiler-vibrant-v0.2.img]
Pulling Singularity image docker://denglab/viroprofiler-host:v0.1 [cache /scratch/project_2007362/software/singularity/cache/denglab-viroprofiler-host-v0.1.img]
Pulling Singularity image docker://denglab/viroprofiler-base:v0.2 [cache /scratch/project_2007362/software/singularity/cache/denglab-viroprofiler-base-v0.2.img]
ERROR ~ Error executing process > 'VIROPROFILER:SETUP:DB_VIRSORTER2'
Caused by:
Failed to pull singularity image
command: singularity pull --name denglab-viroprofiler-virsorter2-v0.2.5.img.pulling.1689159264399 docker://denglab/viroprofiler-virsorter2:v0.2.5 > /dev/null
status : 255
message:
INFO: Converting OCI blobs to SIF format
WARNING: 'nodev' mount option set on /scratch, it could be a source of failure during build process
INFO: Starting build...
Getting image source signatures
Copying blob sha256:7a6db449b51b92eac5c81cdbd82917785343f1664b2be57b22337b0a40c5b29d
Copying blob sha256:b51904ca3834848d576e609bab4b382028643e662db724a58edb9401f9af220c
Copying blob sha256:16f76cf3882a90e2e7969b03b7d0147c189cfb3966d14800091466f2d5fc0a51
Copying blob sha256:377cc88d7c7e949460cc24d07979f2e428f2482e0b360d72860ff1df404208e0
Copying blob sha256:7a15ae76cb488a6adde55d23dfd5571c57bf02206168b7795f7b473b1dc7c218
Copying blob sha256:d0e88a31bdb313b5fab1a325bc79ad5c487cb722dbf65b1561fd4810ebd7b717
Copying blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
Copying blob sha256:301aa5df7f62dcf710fc30e41d585887f918992bc71dba116c7edc267a3c3422
Copying blob sha256:26142b6b2c1ea5de9b7566a9862ea2e9beb02b1a059295cd221ec89c75aac1e8
Copying blob sha256:a7c07239fd155305c23fdbe3cb67838b8d57e0d5922d01b643ec5da4dbf7c7d7
Copying blob sha256:a540153f7d72ad59b0b5d01a2425f0a8aa59bcd9fe6d0784aaeb43301f903aff
Copying blob sha256:c0f68707fa83ac4d248c81ac27d1dd5d378fed88ddf1e6edb8d9fb1f549d75bd
Copying blob sha256:66cbca5e11012c5b367f8b2cf18f81ec91b1780abcc9a34ba008a4ecb197f3a4
Copying config sha256:ef836e11d9dcca820a285d003fb8af9a8e7051282bd5158c1eaa645da4ba00b7
Writing manifest to image destination
Storing signatures
FATAL: While making image from oci registry: error fetching image to cache: while building SIF from layers: conveyor failed to get: Error initializing source oci:/scratch/project_2007362/software/singularity/cache/cache/blob:88d200bb0a32556fa99b4af044d0224b2ddf3c25d05fe56976067b5eaab5eb60: EOF
but at least in this case I have the error to do some tracking.
What do you think?
from emg-viral-pipeline.
Maybe smt is wrong with the singularity image that was pulled and converted for VIRify, causing the latency issue.
Can you remove the singularity images that were already stored for VIRify, and then run the pipeline again with -profile local,singularity
and also define --cores 1 --max_cores 1
. You can use some small test FASTA as input. By that, the pipeline should go through each process step by step and pull the images. Once all the images are stored, you can run the full pipeline.
from emg-viral-pipeline.
Hi,
Still the same error, after a clean installation, and using the parameters you specified. I also think something is wrong with the archiquecture of the HPC. But, how can I figure out it?
from emg-viral-pipeline.
Hm, do you have some admin to ask for help?
Sometimes, on HPC with Singularity, it also helps to specify some environment variables, see here:
export SINGULARITY_TMPDIR=/path/to/some/singularity_tmpdir
E.g. see the discussion here: nextflow-io/nextflow#2786
Or see here https://www.nextflow.io/docs/latest/config.html#config-singularity and scroll down to
Environment variables
Can you try another pipeline I am heavily involved (maybe I then know better what's going on if it also fails or works): https://github.com/replikation/poreCov
nextflow run replikation/poreCov -profile test_fastq,slurm,singularity -r 1.8.3 --update
It might be that you have to define some additional parameters based on your infrastructure. Per default, the pipeline will store downloaded singularity images to a folder ./singularity
and also intermediate files to .work
.
I would be curious if this runs or you get again an error (e.g. the same as for viroprofiler.
from emg-viral-pipeline.
Hi,
I always use the cache dir env variable (its a good practice). Also I tried the new software to test. I got some errors but they are different:
the run code
export SINGULARITY_CACHEDIR="/scratch/sandro/HeP_samples/6_tax_profile/phage/tmpfolder"
nextflow run replikation/poreCov -profile "test_fastq,local,singularity" -r 1.8.3 --max-cores 16 --update --cachedir $SINGULARITY_CACHEDIR
N E X T F L O W ~ version 23.04.1
Launching `https://github.com/replikation/poreCov` [magical_torvalds] DSL2 - revision: b86c57c384 [1.8.3]
________________________________________________________________________________
poreCov | A Nextflow SARS-CoV-2 workflow for nanopore data
Latest available poreCov release: 1.8.3
If neccessary update via: nextflow pull replikation/poreCov
________________________________________________________________________________
Using 4/16 CPU threads per process for a local run [--max_cores]
WARNING: Singularity image building sometimes fails!
Multiple resumes (-resume) and --max_cores 1 --cores 1 for local execution might help.
Warning: Running --update might not be poreCov compatible!
Found latest pangolin container, using: nanozoo/pangolin-v4:4.3--1.21
Found latest nextclade2 container, using: nanozoo/nextclade2:2.14.0--2023-06-16
.
Profile: test_fastq,local,singularity
Current User: valensan
Nextflow-version: 23.04.1
poreCov-version: 1.8.3
Pathing:
Workdir location [-work-Dir]:
/scratch/project_2007362/sandro/HeP_samples/6_tax_profile/phage/work/nextflow-poreCov-valensan
Output dir [--output]:
results
Databases location [--databases]:
/scratch/project_2007362/software/singularity/cache
Singularity cache dir [--cachedir]:
/scratch/project_2007362/software/singularity/cache
______________________________________
Parameters:
Medaka model: r941_min_hac_g507 [--medaka_model]
Min depth nucleotide: 20 [--min_depth]
Latest Pangolin/Nextclade?: true [--update]
CPUs to use: 4 [--cores]
Memory in GB: 12 [--memory]
______________________________________
Primerscheme: V1200 [--primerV]
Min read-length set to: 400 [--minLength]
Max read-length set to: 1500 [--maxLength]
______________________________________
RKI output for german DESH upload:
Output stored at: results/4.RKI-summary
Min Identity to NC_045512.2: 0.90 [--seq_threshold]
Min Depth used: 20 [--min_depth]
Min Depth should be 20 or more for RKI upload
Proportion cutoff N: 0.05 [--n_threshold]
______________________________________
[- ] process > get_nanopore_fastq -
[- ] process > get_nanopore_fastq -
[- ] process > read_qc_wf:nanoplot -
[- ] process > filter_fastq_by_length -
[- ] process > read_classification_wf:down... -
[- ] process > read_classification_wf:kraken2 -
[- ] process > read_classification_wf:krona -
[- ] process > artic_ncov_wf:artic_medaka -
[- ] process > artic_ncov_wf:covarplot -
[- ] process > determine_lineage_wf:pangolin -
[- ] process > determine_mutations_wf:next... -
[- ] process > genome_quality_wf:president -
[- ] process > genome_quality_wf:seqrs -
[- ] process > rki_report_wf:rki_report -
[- ] process > align_to_reference -
[- ] process > create_summary_report_wf:ge... -
[- ] process > create_summary_report_wf:pl... -
[- ] process > create_summary_report_wf:su... -
executor > local (2)
[49/7cd4aa] process > get_nanopore_fastq [ 0%] 0 of 1
[- ] process > read_qc_wf:nanoplot -
[- ] process > filter_fastq_by_length -
[b5/1e2c57] process > read_classification_wf:down... [ 0%] 0 of 1
[- ] process > read_classification_wf:kraken2 -
[- ] process > read_classification_wf:krona -
[- ] process > artic_ncov_wf:artic_medaka -
[- ] process > artic_ncov_wf:covarplot -
[- ] process > determine_lineage_wf:pangolin -
[- ] process > determine_mutations_wf:next... -
[- ] process > genome_quality_wf:president -
[- ] process > genome_quality_wf:seqrs -
[- ] process > rki_report_wf:rki_report -
[- ] process > align_to_reference -
[- ] process > create_summary_report_wf:ge... -
[- ] process > create_summary_report_wf:pl... -
[- ] process > create_summary_report_wf:su... -
executor > local (3)
[49/7cd4aa] process > get_nanopore_fastq [100%] 1 of 1, failed: 1 ✔
[- ] process > read_qc_wf:nanoplot -
[- ] process > filter_fastq_by_length -
[43/c564d7] process > read_classification_wf:down... [ 50%] 1 of 2, failed: 1...
[- ] process > read_classification_wf:kraken2 -
[- ] process > read_classification_wf:krona -
[- ] process > artic_ncov_wf:artic_medaka -
[- ] process > artic_ncov_wf:covarplot -
[- ] process > determine_lineage_wf:pangolin -
[- ] process > determine_mutations_wf:next... -
[- ] process > genome_quality_wf:president -
[- ] process > genome_quality_wf:seqrs -
[- ] process > rki_report_wf:rki_report -
[- ] process > align_to_reference -
[- ] process > create_summary_report_wf:ge... -
[- ] process > create_summary_report_wf:pl... -
[- ] process > create_summary_report_wf:su... -
Not enough reads in all samples, please investigate results/1.Read_quality
Could not generate any genomes, please check your reads results/1.Read_quality
[49/7cd4aa] NOTE: Process `get_nanopore_fastq` terminated with an error exit status (255) -- Error is ignored
[b5/1e2c57] NOTE: Process `read_classification_wf:download_database_kraken2` terminated with an error exit status (255) -- Execution is retried (1)
executor > local (4)
[49/7cd4aa] process > get_nanopore_fastq [100%] 1 of 1, failed: 1 ✔
[- ] process > read_qc_wf:nanoplot -
[- ] process > filter_fastq_by_length -
[43/c564d7] process > read_classification_wf:down... [ 50%] 1 of 2, failed: 1...
[- ] process > read_classification_wf:kraken2 -
[- ] process > read_classification_wf:krona -
[- ] process > artic_ncov_wf:artic_medaka -
[- ] process > artic_ncov_wf:covarplot -
[- ] process > determine_lineage_wf:pangolin -
[- ] process > determine_mutations_wf:next... -
[- ] process > genome_quality_wf:president -
[- ] process > genome_quality_wf:seqrs -
[- ] process > rki_report_wf:rki_report -
[- ] process > align_to_reference -
[4f/832cc7] process > create_summary_report_wf:ge... [100%] 1 of 1, failed: 1 ✔
[- ] process > create_summary_report_wf:pl... -
[- ] process > create_summary_report_wf:su... -
Not enough reads in all samples, please investigate results/1.Read_quality
Could not generate any genomes, please check your reads results/1.Read_quality
[49/7cd4aa] NOTE: Process `get_nanopore_fastq` terminated with an error exit status (255) -- Error is ignored
[b5/1e2c57] NOTE: Process `read_classification_wf:download_database_kraken2` terminated with an error exit status (255) -- Execution is retried (1)
[4f/832cc7] NOTE: Process `create_summary_report_wf:get_variants_classification` terminated with an error exit status (255) -- Error is ignored
slurmstepd: error: *** JOB 17855474 ON r02c22 CANCELLED AT 2023-07-17T13:44:54 DUE TO TIME LIMIT ***
in this case I give the run 30min (which doesn't end).
at least the there are tracking files and it doesn't seems an singularity error.
from emg-viral-pipeline.
Related Issues (20)
- `annotate:ratio_evalue (1)` terminated with an error exit status (1) -- 'nextflow-autodownload-databases' Is a directory error HOT 1
- Can Prodigal-GV make 'real world' difference compared to original Prodigal? HOT 2
- Update ViPhOGs to remove old models that are associated with discontinued viral taxa HOT 5
- CWL discontinued
- No space left in /tmp HOT 2
- Fix sankey plot visualization for undefined ranks
- Imitervirales assignment although not well supported by ViPhOGs
- Repository may be corrupted HOT 3
- Taxonomic ranks are inverted HOT 11
- Issue for running on ARM64-based Linux VM HOT 5
- wrapper_phage_contigs_sorter_iPlant HOT 1
- [Feature Request] Conda Distribution for Taxonomic Identification HOT 1
- Input file name collision on 'annotate:write_gff' HOT 2
- COBRA improves the completeness and contiguity of viral genomes assembled from metagenomes
- Missing process or function with name 'getFileSystem' HOT 3
- Not able to run "nextflow pull EBI-Metagenomics/emg-viral-pipeline" HOT 2
- Error executing process > 'annotate:hmmscan_viphogs (1)' HOT 1
- Trouble to run the example execution - Error executing process > `detect:pprmeta (1)` HOT 6
- Question regarding GFF file generation HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from emg-viral-pipeline.