biocorecrg / mop2 Goto Github PK

View Code? Open in Web Editor NEW

22.0 3.0 7.0 214.06 MB

Master of Pores 2

Home Page: https://biocorecrg.github.io/MOP2/docs/

License: MIT License

Nextflow 71.58% R 2.73% HCL 17.91% Shell 2.64% Perl 5.13%

docker pipeline nanopore singularity nextflow rna cdna rna-seq

mop2's Introduction

MoP2- DSL2 version of Master of Pores

Inspired by Metallica's Master Of Puppets

Install

Please install nextflow and singularity or docker before.

Then download the repo:

git clone --depth 1 --recurse-submodules [email protected]:biocorecrg/MOP2.git

git clone --depth 1 --recurse-submodules https://github.com/biocorecrg/MOP2.git

You can use INSTALL.sh to download the version 3.4.5 of guppy or you can replace it with the version you prefer. Please consider that the support of VBZ compression of fast5 started with version 3.4.X.

cd MOP2; sh INSTALL.sh 3.4.5

Testing

You can replace -with-singularity with -with-docker if you want to use the docker engine.

cd mop_preprocess
nextflow run mop_preprocess.nf -with-singularity -bg -profile local > log

Reference

If you use this tool, please cite our papers:

"Nanopore Direct RNA Sequencing Data Processing and Analysis Using MasterOfPores" Cozzuto L, Delgado-Tejedor A, Hermoso Pulido T, Novoa EM, Ponomarenko J. N. Methods Mol Biol. 2023;2624:185-205. doi: 10.1007/978-1-0716-2962-8_13.

"MasterOfPores: A Workflow for the Analysis of Oxford Nanopore Direct RNA Sequencing Datasets" Luca Cozzuto, Huanle Liu, Leszek P. Pryszcz, Toni Hermoso Pulido, Anna Delgado-Tejedor, Julia Ponomarenko, Eva Maria Novoa. Front. Genet., 17 March 2020. https://doi.org/10.3389/fgene.2020.00211

Documentation

The documentation is available at https://biocorecrg.github.io/MOP2/docs/

mop2's People

Stargazers

Watchers

Forkers

darked89 anaesco toniher codespacebyaditya kyounghyoun matjack2000 max-vdl

mop2's Issues

Can't select genome instead of transcriptome

Hi,
When I try to run MOP2 with "genome" as the reference type instead of "transcriptome" it doesn't create results. But it seems to be running (with the reference genome I provide) using "transcriptome" as the ref_type it seems to be working. The log file doesn't state any errors, in fact, it doesn't provide any lines after:

----------------------CHECK TOOLS -----------------------------
basecalling : guppy

demultiplexing will be skipped
mapping : minimap2
filtering : nanoq
counting : nanocount
discovery will be skipped

[- ] process > flow1:GUPPY_BASECALL:baseCall -
[- ] process > flow1:NANOQ_FILTER:filter -
[- ] process > preprocess_flow:MinIONQC -
[- ] process > preprocess_flow:MINIMAP2:map -
[- ] process > preprocess_flow:SAMTOOLS_CAT:catAln -
[- ] process > preprocess_flow:SAMTOOLS_SORT:sortAln -
[- ] process > preprocess_flow:SAMTOOLS_INDEX:indexBam -
[- ] process > preprocess_flow:checkRef -

permission denied

Hi,
When I try to do
git clone --depth 1 --recurse-submodules [email protected]:biocorecrg/MOP2.git

I get
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

I have tried this on a linux machine and a mac with the same issue

Bug identifying files

I think there is bug where the software cannot identify the fast5 or fastq files it is given. When I run MOP2, I get:

Argument of file function cannot be empty

-- Check script '../BioNextflow/subworkflows/read_count/htseq.nf' at line: 65 or see '.nextflow.log' file for more details
Cannot find any file matching: /path/.fastq
If I do "ls /path/.fastq" it will identify the files but the pipeliner can't and gives the error above.

I appreciate your help

Run mop_preprocess with Apple M1 chip

Hi,

I am running the mop_preprocess workflow in a computer with an Apple M1 chip. I am getting an error related to it. Is there any way to circumvent it? The workflow is completed successfully when I include just a couple of input file, but it crashes when I include the entire set of files I need to analyze.

Thanks!
Jesús.

N E X T F L O W ~ version 22.04.0
Launching mop_preprocess.nf [sad_majorana] DSL2 - revision: ec40fe0af4

╔╦╗╔═╗╔═╗ ╔═╗┬─┐┌─┐┌─┐┬─┐┌─┐┌─┐┌─┐┌─┐┌─┐
║║║║ ║╠═╝ ╠═╝├┬┘├┤ ├─┘├┬┘│ ││ ├┤ └─┐└─┐
╩ ╩╚═╝╩ ╩ ┴└─└─┘┴ ┴└─└─┘└─┘└─┘└─┘└─┘

====================================================
BIOCORE@CRG Master of Pores 2. Preprocessing - N F ~ version 2.0

conffile. : /Users/j/MOP2/data/W/final_summary_FAR92050_1b93b532.txt

fast5 :
fastq : /Users/j/MOP2/data/W/fastq_pass/*.fastq.gz

reference : /Users/j/MOP2/anno/At.fa
annotation :

granularity. : 5

ref_type : transcriptome
pars_tools : drna_tool_splice_opt.tsv

output : /Users/j/MOP2/output/

GPU : ON

basecalling : NO
demultiplexing : NO
demulti_fast5 : NO

filtering : nanoq
mapping : minimap2

counting : nanocount
discovery : bambu

cram_conv : NO
subsampling_cram : NO

saveSpace : NO
email :

Skipping the email

----------------------CHECK TOOLS -----------------------------

basecalling will be skipped
demultiplexing will be skipped
mapping : minimap2
filtering : nanoq
counting : nanocount
discovery : bambu

[- ] process > preprocess_simple:FASTQC:fa... -

[- ] process > preprocess_simple:FASTQC:fa... -
[- ] process > preprocess_simple:MINIMAP2:map -
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:bam2stats -
[- ] process > preprocess_simple:joinAlnStats -
[- ] process > preprocess_simple:NANOPLOT_... -
[- ] process > preprocess_simple:NANOCOUNT... -
[- ] process > preprocess_simple:AssignReads -
[- ] process > preprocess_simple:countStats -
[- ] process > preprocess_simple:joinCount... -
[- ] process > preprocess_simple:MULTIQC:m... -

[- ] process > preprocess_simple:FASTQC:fa... [ 0%] 0 of 20
[- ] process > preprocess_simple:MINIMAP2:map [ 0%] 0 of 10
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:bam2stats -
[- ] process > preprocess_simple:joinAlnStats -
[- ] process > preprocess_simple:NANOPLOT_... -
[- ] process > preprocess_simple:NANOCOUNT... -
[- ] process > preprocess_simple:AssignReads -
[- ] process > preprocess_simple:countStats -
[- ] process > preprocess_simple:joinCount... -
[- ] process > preprocess_simple:MULTIQC:m... -

executor > local (2)
[f5/7a6bac] process > preprocess_simple:FASTQC:fa... [ 0%] 0 of 41
[- ] process > preprocess_simple:MINIMAP2:map [ 0%] 0 of 36
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:bam2stats -
[- ] process > preprocess_simple:joinAlnStats -
[- ] process > preprocess_simple:NANOPLOT_... -
[- ] process > preprocess_simple:NANOCOUNT... -
[- ] process > preprocess_simple:AssignReads -
[- ] process > preprocess_simple:countStats -
[- ] process > preprocess_simple:joinCount... -
[- ] process > preprocess_simple:MULTIQC:m... -

executor > local (9)
[d9/74fd83] process > preprocess_simple:FASTQC:fa... [ 0%] 0 of 60
[- ] process > preprocess_simple:MINIMAP2:map [ 0%] 0 of 53
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:bam2stats -
[- ] process > preprocess_simple:joinAlnStats -
[- ] process > preprocess_simple:NANOPLOT_... -
[- ] process > preprocess_simple:NANOCOUNT... -
[- ] process > preprocess_simple:AssignReads -
[- ] process > preprocess_simple:countStats -
[- ] process > preprocess_simple:joinCount... -
[- ] process > preprocess_simple:MULTIQC:m... -

executor > local (10)
[7b/80748a] process > preprocess_simple:FASTQC:fa... [ 0%] 0 of 72
[- ] process > preprocess_simple:MINIMAP2:map [ 0%] 0 of 64
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:bam2stats -
[- ] process > preprocess_simple:joinAlnStats -
[- ] process > preprocess_simple:NANOPLOT_... -
[- ] process > preprocess_simple:NANOCOUNT... -
[- ] process > preprocess_simple:AssignReads -
[- ] process > preprocess_simple:countStats -
[- ] process > preprocess_simple:joinCount... -
[- ] process > preprocess_simple:MULTIQC:m... -

executor > local (10)
[7b/80748a] process > preprocess_simple:FASTQC:fa... [ 0%] 0 of 86
[- ] process > preprocess_simple:MINIMAP2:map [ 0%] 0 of 81
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:bam2stats -
[- ] process > preprocess_simple:joinAlnStats -
[- ] process > preprocess_simple:NANOPLOT_... -
[- ] process > preprocess_simple:NANOCOUNT... -
[- ] process > preprocess_simple:AssignReads -
[- ] process > preprocess_simple:countStats -
[- ] process > preprocess_simple:joinCount... -
[- ] process > preprocess_simple:MULTIQC:m... -

executor > local (10)
[7b/80748a] process > preprocess_simple:FASTQC:fa... [ 0%] 0 of 118
[- ] process > preprocess_simple:MINIMAP2:map [ 0%] 0 of 104
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:bam2stats -
[- ] process > preprocess_simple:joinAlnStats -
[- ] process > preprocess_simple:NANOPLOT_... -
[- ] process > preprocess_simple:NANOCOUNT... -
[- ] process > preprocess_simple:AssignReads -
[- ] process > preprocess_simple:countStats -
[- ] process > preprocess_simple:joinCount... -
[- ] process > preprocess_simple:MULTIQC:m... -

executor > local (10)
[7b/80748a] process > preprocess_simple:FASTQC:fa... [ 0%] 0 of 178
[- ] process > preprocess_simple:MINIMAP2:map [ 0%] 0 of 168
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:bam2stats -
[- ] process > preprocess_simple:joinAlnStats -
[- ] process > preprocess_simple:NANOPLOT_... -
[- ] process > preprocess_simple:NANOCOUNT... -
[- ] process > preprocess_simple:AssignReads -
[- ] process > preprocess_simple:countStats -
[- ] process > preprocess_simple:joinCount... -
[- ] process > preprocess_simple:MULTIQC:m... -

executor > local (10)
[7b/80748a] process > preprocess_simple:FASTQC:fa... [ 0%] 0 of 185
[- ] process > preprocess_simple:MINIMAP2:map [ 0%] 0 of 175
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:bam2stats -
[- ] process > preprocess_simple:joinAlnStats -
[- ] process > preprocess_simple:NANOPLOT_... -
[- ] process > preprocess_simple:NANOCOUNT... -
[- ] process > preprocess_simple:AssignReads -
[- ] process > preprocess_simple:countStats -
[- ] process > preprocess_simple:joinCount... -
[- ] process > preprocess_simple:MULTIQC:m... -

executor > local (10)
[7b/80748a] process > preprocess_simple:FASTQC:fa... [ 0%] 0 of 198
[- ] process > preprocess_simple:MINIMAP2:map [ 0%] 0 of 186
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:bam2stats -
[- ] process > preprocess_simple:joinAlnStats -
[- ] process > preprocess_simple:NANOPLOT_... -
[- ] process > preprocess_simple:NANOCOUNT... -
[- ] process > preprocess_simple:AssignReads -
[- ] process > preprocess_simple:countStats -
[- ] process > preprocess_simple:joinCount... -
[- ] process > preprocess_simple:MULTIQC:m... -

executor > local (10)
[7b/80748a] process > preprocess_simple:FASTQC:fa... [ 0%] 0 of 208
[- ] process > preprocess_simple:MINIMAP2:map [ 0%] 0 of 196
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:bam2stats -
[- ] process > preprocess_simple:joinAlnStats -
[- ] process > preprocess_simple:NANOPLOT_... -
[- ] process > preprocess_simple:NANOCOUNT... -
[- ] process > preprocess_simple:AssignReads -
[- ] process > preprocess_simple:countStats -
[- ] process > preprocess_simple:joinCount... -
[- ] process > preprocess_simple:MULTIQC:m... -

executor > local (10)
[7b/80748a] process > preprocess_simple:FASTQC:fa... [ 0%] 0 of 210
[- ] process > preprocess_simple:MINIMAP2:map [ 0%] 0 of 201
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:bam2stats -
[- ] process > preprocess_simple:joinAlnStats -
[- ] process > preprocess_simple:NANOPLOT_... -
[- ] process > preprocess_simple:NANOCOUNT... -
[- ] process > preprocess_simple:AssignReads -
[- ] process > preprocess_simple:countStats -
[- ] process > preprocess_simple:joinCount... -
[- ] process > preprocess_simple:MULTIQC:m... -

executor > local (10)
[7b/80748a] process > preprocess_simple:FASTQC:fa... [ 0%] 0 of 231
[- ] process > preprocess_simple:MINIMAP2:map [ 0%] 0 of 218
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:bam2stats -
[- ] process > preprocess_simple:joinAlnStats -
[- ] process > preprocess_simple:NANOPLOT_... -
[- ] process > preprocess_simple:NANOCOUNT... -
[- ] process > preprocess_simple:AssignReads -
[- ] process > preprocess_simple:countStats -
[- ] process > preprocess_simple:joinCount... -
[- ] process > preprocess_simple:MULTIQC:m... -

executor > local (10)
[7b/80748a] process > preprocess_simple:FASTQC:fa... [ 0%] 0 of 252
[- ] process > preprocess_simple:MINIMAP2:map [ 0%] 0 of 233
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:bam2stats -
[- ] process > preprocess_simple:joinAlnStats -
[- ] process > preprocess_simple:NANOPLOT_... -
[- ] process > preprocess_simple:NANOCOUNT... -
[- ] process > preprocess_simple:AssignReads -
[- ] process > preprocess_simple:countStats -
[- ] process > preprocess_simple:joinCount... -
[- ] process > preprocess_simple:MULTIQC:m... -

executor > local (10)
[7b/80748a] process > preprocess_simple:FASTQC:fa... [ 0%] 0 of 270
[- ] process > preprocess_simple:MINIMAP2:map [ 0%] 0 of 259
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:bam2stats -
[- ] process > preprocess_simple:joinAlnStats -
[- ] process > preprocess_simple:NANOPLOT_... -
[- ] process > preprocess_simple:NANOCOUNT... -
[- ] process > preprocess_simple:AssignReads -
[- ] process > preprocess_simple:countStats -
[- ] process > preprocess_simple:joinCount... -
[- ] process > preprocess_simple:MULTIQC:m... -

executor > local (10)
[7b/80748a] process > preprocess_simple:FASTQC:fa... [ 0%] 0 of 296
[- ] process > preprocess_simple:MINIMAP2:map [ 0%] 0 of 283
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:bam2stats -
[- ] process > preprocess_simple:joinAlnStats -
[- ] process > preprocess_simple:NANOPLOT_... -
[- ] process > preprocess_simple:NANOCOUNT... -
[- ] process > preprocess_simple:AssignReads -
[- ] process > preprocess_simple:countStats -
[- ] process > preprocess_simple:joinCount... -
[- ] process > preprocess_simple:MULTIQC:m... -

executor > local (10)
[7b/80748a] process > preprocess_simple:FASTQC:fa... [ 0%] 0 of 323
[- ] process > preprocess_simple:MINIMAP2:map [ 0%] 0 of 323
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:bam2stats -
[- ] process > preprocess_simple:joinAlnStats -
[- ] process > preprocess_simple:NANOPLOT_... -
[- ] process > preprocess_simple:NANOCOUNT... -
[- ] process > preprocess_simple:AssignReads -
[- ] process > preprocess_simple:countStats -
[- ] process > preprocess_simple:joinCount... -
[- ] process > preprocess_simple:MULTIQC:m... -

executor > local (11)
[7b/80748a] process > preprocess_simple:FASTQC:fa... [ 0%] 0 of 323
[fd/bc4a52] process > preprocess_simple:MINIMAP2:... [ 0%] 0 of 323
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:bam2stats -
[- ] process > preprocess_simple:joinAlnStats -
[- ] process > preprocess_simple:NANOPLOT_... -
[- ] process > preprocess_simple:NANOCOUNT... -
[- ] process > preprocess_simple:AssignReads -
[- ] process > preprocess_simple:countStats -
[- ] process > preprocess_simple:joinCount... -
[- ] process > preprocess_simple:MULTIQC:m... -

executor > local (12)
[7f/54b157] process > preprocess_simple:FASTQC:fa... [ 0%] 2 of 323
[44/96cc2e] process > preprocess_simple:MINIMAP2:... [ 0%] 0 of 323
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:bam2stats -
[- ] process > preprocess_simple:joinAlnStats -
[- ] process > preprocess_simple:NANOPLOT_... -
[- ] process > preprocess_simple:NANOCOUNT... -
[- ] process > preprocess_simple:AssignReads -
[- ] process > preprocess_simple:countStats -
[- ] process > preprocess_simple:joinCount... -
[- ] process > preprocess_simple:MULTIQC:m... -

executor > local (13)
[f2/0e9713] process > preprocess_simple:FASTQC:fa... [ 0%] 3 of 323
[a6/dc7f07] process > preprocess_simple:MINIMAP2:... [ 0%] 0 of 323
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:bam2stats -
[- ] process > preprocess_simple:joinAlnStats -
[- ] process > preprocess_simple:NANOPLOT_... -
[- ] process > preprocess_simple:NANOCOUNT... -
[- ] process > preprocess_simple:AssignReads -
[- ] process > preprocess_simple:countStats -
[- ] process > preprocess_simple:joinCount... -
[- ] process > preprocess_simple:MULTIQC:m... -

executor > local (14)
[bf/fb552b] process > preprocess_simple:FASTQC:fa... [ 1%] 4 of 323
[80/aae8d2] process > preprocess_simple:MINIMAP2:... [ 0%] 0 of 323
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:bam2stats -
[- ] process > preprocess_simple:joinAlnStats -
[- ] process > preprocess_simple:NANOPLOT_... -
[- ] process > preprocess_simple:NANOCOUNT... -
[- ] process > preprocess_simple:AssignReads -
[- ] process > preprocess_simple:countStats -
[- ] process > preprocess_simple:joinCount... -
[- ] process > preprocess_simple:MULTIQC:m... -

executor > local (15)
[7b/80748a] process > preprocess_simple:FASTQC:fa... [ 1%] 5 of 323
[df/14a7ec] process > preprocess_simple:MINIMAP2:... [ 0%] 0 of 323
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:bam2stats -
[- ] process > preprocess_simple:joinAlnStats -
[- ] process > preprocess_simple:NANOPLOT_... -
[- ] process > preprocess_simple:NANOCOUNT... -
[- ] process > preprocess_simple:AssignReads -
[- ] process > preprocess_simple:countStats -
[- ] process > preprocess_simple:joinCount... -
[- ] process > preprocess_simple:MULTIQC:m... -

executor > local (16)
[f5/7a6bac] process > preprocess_simple:FASTQC:fa... [ 1%] 6 of 323
[98/1e7dce] process > preprocess_simple:MINIMAP2:... [ 0%] 0 of 323
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:bam2stats -
[- ] process > preprocess_simple:joinAlnStats -
[- ] process > preprocess_simple:NANOPLOT_... -
[- ] process > preprocess_simple:NANOCOUNT... -
[- ] process > preprocess_simple:AssignReads -
[- ] process > preprocess_simple:countStats -
[- ] process > preprocess_simple:joinCount... -
[- ] process > preprocess_simple:MULTIQC:m... -

executor > local (17)
[f6/65194f] process > preprocess_simple:FASTQC:fa... [ 2%] 7 of 323
[f9/242a42] process > preprocess_simple:MINIMAP2:... [ 0%] 0 of 323
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:bam2stats -
[- ] process > preprocess_simple:joinAlnStats -
[- ] process > preprocess_simple:NANOPLOT_... -
[- ] process > preprocess_simple:NANOCOUNT... -
[- ] process > preprocess_simple:AssignReads -
[- ] process > preprocess_simple:countStats -
[- ] process > preprocess_simple:joinCount... -
[- ] process > preprocess_simple:MULTIQC:m... -

executor > local (19)
[20/952b18] process > preprocess_simple:FASTQC:fa... [ 2%] 9 of 323
[8f/41e61d] process > preprocess_simple:MINIMAP2:... [ 0%] 0 of 323
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:bam2stats -
[- ] process > preprocess_simple:joinAlnStats -
[- ] process > preprocess_simple:NANOPLOT_... -
[- ] process > preprocess_simple:NANOCOUNT... -
[- ] process > preprocess_simple:AssignReads -
[- ] process > preprocess_simple:countStats -
[- ] process > preprocess_simple:joinCount... -
[- ] process > preprocess_simple:MULTIQC:m... -

executor > local (20)
[b2/64f673] process > preprocess_simple:FASTQC:fa... [ 3%] 10 of 323
[bb/6366af] process > preprocess_simple:MINIMAP2:... [ 0%] 0 of 323
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:bam2stats -
[- ] process > preprocess_simple:joinAlnStats -
[- ] process > preprocess_simple:NANOPLOT_... -
[- ] process > preprocess_simple:NANOCOUNT... -
[- ] process > preprocess_simple:AssignReads -
[- ] process > preprocess_simple:countStats -
[- ] process > preprocess_simple:joinCount... -
[- ] process > preprocess_simple:MULTIQC:m... -

executor > local (21)
[cf/17126d] process > preprocess_simple:FASTQC:fa... [ 3%] 10 of 323
[bb/6366af] process > preprocess_simple:MINIMAP2:... [ 0%] 0 of 323
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:bam2stats -
[- ] process > preprocess_simple:joinAlnStats -
[- ] process > preprocess_simple:NANOPLOT_... -
[- ] process > preprocess_simple:NANOCOUNT... -
[- ] process > preprocess_simple:AssignReads -
[- ] process > preprocess_simple:countStats -
[- ] process > preprocess_simple:joinCount... -
[- ] process > preprocess_simple:MULTIQC:m... -
Error executing process > 'preprocess_simple:MINIMAP2:map (FAR92050_pass_1b93b532_287)'

Caused by:
Process preprocess_simple:MINIMAP2:map (FAR92050_pass_1b93b532_287) terminated with an error exit status (1)

Command executed:

minimap2 -t 1 -a -uf -ax splice -k14 At.fa FAR92050_pass_1b93b532_287.fastq.gz | samtools view -@ 1 -F4 -hSb - > FAR92050_pass_1b93b532_287.bam

Command exit status:
1

Command output:
(empty)

Command error:
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
[M::mm_idx_gen::29.912*0.65] collected minimizers
[main_samview] fail to read the header from "-".

Work dir:
/Users/j/MOP2/mop_preprocess/work/98/1e7dceeb1693b263391ae6cd69f897

Tip: when you have fixed the problem you can continue the execution adding the option -resume to the run command line

Pipeline BIOCORE@CRG Master of Pore - preprocess completed!
Started at 2022-06-01T12:32:24.699108+02:00
Finished at 2022-06-01T12:34:35.078982+02:00
Time elapsed: 2m 10s
Execution status: failed
WARN: Killing running tasks (9)

executor > local (21)
[cf/17126d] process > preprocess_simple:FASTQC:fa... [ 3%] 10 of 323
[bb/6366af] process > preprocess_simple:MINIMAP2:... [ 1%] 4 of 322, failed: 4
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:SAMTOOLS_... -
[- ] process > preprocess_simple:bam2stats -
[- ] process > preprocess_simple:joinAlnStats -
[- ] process > preprocess_simple:NANOPLOT_... -
[- ] process > preprocess_simple:NANOCOUNT... -
[- ] process > preprocess_simple:AssignReads -
[- ] process > preprocess_simple:countStats -
[- ] process > preprocess_simple:joinCount... -
[- ] process > preprocess_simple:MULTIQC:m... -
Error executing process > 'preprocess_simple:MINIMAP2:map (FAR92050_pass_1b93b532_287)'

Caused by:
Process preprocess_simple:MINIMAP2:map (FAR92050_pass_1b93b532_287) terminated with an error exit status (1)

Command executed:

minimap2 -t 1 -a -uf -ax splice -k14 At.fa FAR92050_pass_1b93b532_287.fastq.gz | samtools view -@ 1 -F4 -hSb - > FAR92050_pass_1b93b532_287.bam

Command exit status:
1

Command output:
(empty)

Work dir:
/Users/jj/MOP2/mop_preprocess/work/98/1e7dceeb1693b263391ae6cd69f897

Tip: when you have fixed the problem you can continue the execution adding the option -resume to the run command line

INSTALL.sh update links

When rerunning (to update guppy version for example) INSTALL.sh, it will fail when creating symlinks to ont-guppy bins and libs.

Maybe it could be updated to something like (to force symlinks)

ln -sf ont-guppy/bin/guppy_* .
ln -sf ont-guppy/lib/* .

Similar for the move operation with something like

rm -rf mop_preprocess/bin/ont-guppy_old
mv mop_preprocess/bin/ont-guppy mop_preprocess/bin/ont-guppy_old
mv ont-guppy mop_preprocess/bin/

upgrade nanocount

Nanocount has a bug. It needs to be updated

RE: running mop_preprocessing on multiple runs

Hi,
I have three minion runs of dRNA that I am trying to analyze using MOP2.

I understand that it is possible to run mop_prerocessing on multiple runs. I am not clear on how to input the kit and flow cell info though. in the docs, it says they can be inputed in one of the _opt.tsv file. I used the following:
Library prep kit: SQK-RNA002
Flowcell: FLO-MIN106
Is this all info that I need to add to one of the _opt.tsv files? I yes, what formate?

alternatively, I have the "final_summary.txt" for the three runs. can I cat them in one file and pass it to the confine option?

I had base calling done during the sequencing run. So, I have both fast5_fail and fast5_pass. I understand the the this mop_preporcessing with do the base calling. so, can I just combine both passed and failed fast5 in one file per run and pass them to the workflow?
there are three dRNA *_opt.tsv files.

drna_tool_splice_opt.tsv
drna_tool_unsplice_guppy6_opt.tsv
drna_tool_unsplice_opt.tsv

I am clear on when to use the splice or unsplice parameters. what exactly is the difference? also, can the flag --disable_qscore_filtering be passed to guppy in the splice parameter file?

Many thanks,

mop_mod running time

Salutations,

First of all, thank you very much for your great and important work.
I have a few questions. I am running Master of Pores 2 on a workstation from Nanopore Promethion (370 gb ram, 112 cores). The first stage, mop_preprocess.nf finished pretty quickly. What cannot be said about the second stage - mop_mod.nf.

(mop2) prom@PC48A067:/data/adaniyarov/directRNA_DRS/data_Control_P2_EpiNano/mop2/MOP2/mop_mod$ 
/data/adaniyarov/directRNA_DRS/data_Control_P2_EpiNano/mop2/nextflow 
run /data/adaniyarov/directRNA_DRS/data_Control_P2_EpiNano/mop2/MOP2/mop_mod/mop_mod.nf 
-with-singularity -profile standard
N E X T F L O W  ~  version 22.10.2
Launching `/data/adaniyarov/directRNA_DRS/data_Control_P2_EpiNano/mop2/MOP2/mop_mod/mop_mod.nf` [naughty_bassi] DSL2 - revision: 83320fa996


╔╦╗╔═╗╔═╗  ╔╦╗┌─┐┌┬┐
║║║║ ║╠═╝  ║║║│ │ ││
╩ ╩╚═╝╩    ╩ ╩└─┘─┴┘
                                                                                       
====================================================
BIOCORE@CRG Master of Pores 2. Detection of RNA modification - N F  ~  version 2.0
====================================================

*****************   Input files    *******************
input_path                              : /path/mop2/MOP2/mop_preprocess/output_1_1_fast5/
comparison                              : /path/mop2/MOP2/mop_mod/comparison.tsv

********** reference has to be the genome *************
reference                               : /data/PublicData/refSeq_transcripts/GRCh38_latest_rna.fna
output                                  : /path/mop2/MOP2/mop_mod/output_mod

pars_tools				: /path/mop2/MOP2/mop_mod/tools_opt.tsv

************************* Flows *******************************
epinano                             	: YES
nanocompore                             : NO
tombo_lsc                               : YES
tombo_msc                               : YES

email                                   : 

Skipping the email

executor >  local (19994)
[61/36fba8] process > checkRef (Checking GRCh38_latest_rna.fna)                                           [100%] 1 of 1 ✔
[c0/d1417d] process > epinano_flow:splitReference (Splitting of reference.fa)                             [100%] 1 of 1 ✔
[b1/21ecdc] process > epinano_flow:splitBams (Splitting of wt_s.bam on pieces999.fa)                      [100%] 13320 of 13320 ✔
[96/675a4a] process > epinano_flow:indexReference (Indexing pieces999.fa)                                 [100%] 6660 of 6660 ✔
[81/b93824] process > epinano_flow:EPINANO_CALC_VAR_FREQUENCIES (wt___pieces01_s.bam on wt)               [  0%] 0 of 13320
[-        ] process > epinano_flow:joinEpinanoRes                                                         -
[-        ] process > epinano_flow:makeEpinanoPlots_ins                                                   -
[-        ] process > epinano_flow:makeEpinanoPlots_mis                                                   -
[-        ] process > epinano_flow:makeEpinanoPlots_del                                                   -
[7b/56cd78] process > tombo_common_flow:multiToSingleFast5 (wt___PAI52977_pass_d5246539_0)                [100%] 2 of 2 ✔
[5d/6c2b2c] process > tombo_common_flow:TOMBO_RESQUIGGLE_RNA:resquiggle_rna (mod___PAI53910_pass_e5b17... [100%] 2 of 2 ✔
[70/ecd689] process > getChromInfo (reference.fa)                                                         [100%] 1 of 1 ✔
[1c/0082b5] procestombo_msc_flow:TOMBO_GET_MODIFICATION_MSC:getModificationsWithModelSampleCompare... [100%] 1 of 1 ✔
[-        ] process > bedGraphToWig_msc                                                                   [  0%] 0 of 4
[75/9ce51c] process > tombo_lsc_flow:TOMBO_GET_MODIFICATION_LSC:getModificationsWithLevelSampleCompare... [100%] 1 of 1 ✔
[-        ] process > bedGraphToWig_lsc                                                                   [  0%] 0 of 4
[-        ] process > wigToBigWig                                                                         [  0%] 0 of 4
[-        ] process > mergeTomboWigsPlus                                                                  -
[-        ] process > mergeTomboWigsMinus                                                                 -
[01/de5629] process > EPINANO_VER:getVersion                                                              [100%] 1 of 1 ✔
[32/7baa16] process > NANOPOLISH_VER:getVersion                                                           [100%] 1 of 1 ✔
[d6/512a0f] process > NANOCOMPORE_VER:getVersion                                                          [100%] 1 of 1 ✔
[7e/1a182b] process > TOMBO_VER:getVersion                                                                [100%] 1 of 1 ✔

The stage of searching for modifications (mop_mod.nf) has been going on for almost 24 hours, no signs of work are visible. Htop shows no load on the server.

process {
  cpus = 110
  memory='350G'
  cache='lenient'
  container = 'biocorecrg/mopprepr:0.7'
  containerOptions = { workflow.containerEngine == "docker" ? '-u $(id -u):$(id -g)': null}

  withLabel: big_cpus {
		cpus = 80
		memory = '80G'
  }
  withLabel: big_cpus_ignore {
        errorStrategy = 'ignore'
        cpus = 8
        memory = '0G'
  } 
  withLabel: big_mem_cpus {
		time = '30h'
		cpus = 50
		memory = '300G'
  }
  withLabel: basecall_cpus {
		cpus = 80
		memory = '80G'
  }
  withLabel: basecall_gpus {
		maxForks = 1
		memory = '80G'
	containerOptions = { workflow.containerEngine == "singularity" ? '--nv':
		   ( workflow.containerEngine == "docker" ? '-u $(id -u):$(id -g) --gpus all': null ) } 
  }
}

I tried it with -profile standard and -profile local. No change.

process {
	executor = 'local'
	cpus = 100
	memory = '300GB'    
    cache='lenient'
    container = 'biocorecrg/mopprepr:0.7'
    containerOptions = { workflow.containerEngine == "docker" ? '-u $(id -u):$(id -g)': null}
    withLabel: big_cpus_ignore {
        errorStrategy = 'ignore'
	
    }
    withLabel: basecall_gpus {
	    maxForks = 1
	    containerOptions = { workflow.containerEngine == "singularity" ? '--nv':
		   ( workflow.containerEngine == "docker" ? '-u $(id -u):$(id -g) --gpus all': null ) } 
    }
}

Could you please advise how to properly configure the configurations to get the work done as quickly as possible?

(base) prom@PC48A067:~$ lscpu
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   46 bits physical, 48 bits virtual
CPU(s):                          112
On-line CPU(s) list:             0-111
Thread(s) per core:              2
Core(s) per socket:              28
Socket(s):                       2
NUMA node(s):                    2
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           85
Model name:                      Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz
Stepping:                        4
CPU MHz:                         1000.000
CPU max MHz:                     3800.0000
CPU min MHz:                     1000.0000
BogoMIPS:                        5000.00
Virtualization:                  VT-x
L1d cache:                       1.8 MiB
L1i cache:                       1.8 MiB
L2 cache:                        56 MiB
L3 cache:                        77 MiB
NUMA node0 CPU(s):               0-27,56-83
NUMA node1 CPU(s):               28-55,84-111

(base) prom@PC48A067:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:          376Gi       7.7Gi       343Gi       247Mi        25Gi       366Gi
Swap:          76Gi          0B        76Gi

There are a few more questions about using references (genome, transcriptome) and running Master of Pores 2 on another machine. I will open another discussion. Thanks.

Error when running mop_mod

Hi,

I have been trying these few weeks (at different time) and still get the same error.

Error executing process > 'compore_polish_flow:NANOPOLISH_EVENTALIGN:index (wt)'

Caused by:
  Failed to pull singularity image
  command: singularity pull  --name biocorecrg-mopnanopolish-0.2.img.pulling.1683256214123 docker://biocorecrg/mopnanopolish:0.2 > /dev/null
  status : 255
  message:
    FATAL:   While making image from oci registry: error fetching image to cache: failed to get checksum for docker://biocorecrg/mopnanopolish:0.2: pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp 52.1.184.176:443: i/o timeout

Is there anything I did wrong?
MOP2_mod_log_202305051110.txt

Error with Guppy Basecall: cat: 'fast5_fail---7_out/*.fastq': No such file or directory

Hello,

I'm running MOP2 on a Macintosh computer with Nextflow 21.10.6 build 5660 and Guppy 6.0.1. MOP2 will run if I start it with the fastq files, but if I try to run basecalling with fast5 files it fails. I get the following error:

executor >  local (3)
[e4/54a1e0] process > flow1:GUPPY_BASECALL:baseCall (fast5_fail---2)            [  0%] 1 of 329, failed: 1
[-        ] process > flow1:NANOQ_FILTER:filter                                 -
[-        ] process > preprocess_flow:MinIONQC                                  -
[-        ] process > preprocess_flow:GRAPHMAP2:map                             -
[-        ] process > preprocess_flow:SAMTOOLS_CAT:catAln                       -
[-        ] process > preprocess_flow:SAMTOOLS_SORT:sortAln                     -
[-        ] process > preprocess_flow:SAMTOOLS_INDEX:indexBam                   -
[c8/cc31ee] process > preprocess_flow:checkRef (Checking GRCh38.primary_asse... [100%] 1 of 1 ✔
[-        ] process > preprocess_flow:bam2Cram                                  -
[-        ] process > preprocess_flow:bam2stats                                 -
[-        ] process > preprocess_flow:joinAlnStats                              -
[-        ] process > preprocess_flow:NANOPLOT_QC:MOP_nanoPlot                  -
[-        ] process > preprocess_flow:concatenateFastQFiles                     -
[-        ] process > preprocess_flow:FASTQC:fastQC                             -
[-        ] process > preprocess_flow:MULTIQC:makeReport                        -
Error executing process > 'flow1:GUPPY_BASECALL:baseCall (fast5_fail---7)

Caused by:
  Process 'flow1:GUPPY_BASECALL:baseCall (fast5_fail---7)' terminated with an error exit status (1)

Command executed:

  guppy_basecaller          --fast5_out --flowcell FLO-MIN106 --kit SQK-RNA002  -i ./         --save_path ./fast5_fail---7_out         --gpu_runners_per_device 1         --cpu_threads_per_caller 1 	    --num_callers  8
  cat fast5_fail---7_out/*.fastq >> fast5_fail---7.fastq
  rm fast5_fail---7_out/*.fastq
  gzip fast5_fail---7.fastq

Command exit status:
  1

Command output:
  ONT Guppy basecalling software version 6.0.1+652ffd179
  config file:        /Users/tj/MOP2/mop_preprocess/bin/ont-guppy/data/rna_r9.4.1_70bps_hac.cfg
  model file:         /Users/tj/MOP2/mop_preprocess/bin/ont-guppy/data/template_rna_r9.4.1_70bps_hac.jsn
  input path:         ./
  save path:          ./fast5_fail---7_out
  chunk size:         2000
  chunks per runner:  512
  minimum qscore:     7
  records per file:   4000
  num basecallers:    8
  cpu mode:           ON
  threads per caller: 1
  
  Found 1 fast5 files to process.
  Init time: 202 ms
  
  0%   10   20   30   40   50   60   70   80   90   100%
  |----|----|----|----|----|----|----|----|----|----|
  ***************************************************
  Caller time: 231 ms, Samples called: 0, samples/s: 0
  Finishing up any open output files.
  Basecalling completed successfully.

Command error:
  cat: 'fast5_fail---7_out/*.fastq': No such file or directory

Work dir:
  /Users/tj/MOP2/mop_preprocess/work/be/896b2b971a64d7483d9121934582ac

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named '.command.sh'

My params.config file is as follows:

params {
    conffile            = "/Users/tj/MOP2/data/U87_WT_122021/final_summary_FAR91556_a92bf75a.txt"
    fast5               = "/Users/tj/MOP2/data/U87_WT_122021/**/*.fast5"
    fastq               = ""

    reference           = "/Users/tj/MOP2/references/GRCh38.primary_assembly.genome.fa"
    annotation          = "/Users/tj/MOP2/anno/gencode.v35.annotation.gtf"
    ref_type            = "genome"

    pars_tools          = "drna_tool_splice_opt.tsv" 
    output              = "/Users/tj/MOP2/output/preprocess/U87_WT_122021"
    qualityqc           = 1
    granularity         = 1

    basecalling         = "guppy"
    GPU                 = "OFF"
    demultiplexing      = "NO"
    demulti_fast5       = "NO" 

    filtering           = "nanoq"

    mapping             = "minimap2"
    counting            = "NO"
    discovery           = "NO"

    cram_conv           = "YES"
    subsampling_cram    = 50

    saveSpace           = "NO"

    email               = ""
}

I notice that my error is similar to the one reported here, and so I think Guppy is looking for files in the wrong location? I tried troubleshooting on my own and doing an intensive google search, but I am stuck.

Thank you so much!

Inverted results in polya_common/fast5_pass_joined.txt

Hi! Thank you for developing such a convenient workflow for ONT sequencing. But I encountered a small issue after running polyA module. It seems that Nanopolish and Tailfindr results were assigned to incorrect columns in jointed text. Maybe it was related to the incorrect parameter parsing order below?

MOP2/local_modules.nf

Line 698 in 9469ca8

tuple val(sampleID), path(nanopol), path(tailfindr), path(genes)

I am not quite sure if this was my mistake. If the script was correct, please close this issue and sorry for bothering.
Sincerely, Yuancun

deeplexicon final table in nextflow output

Hi Luca, could we save deeplexicon table (.tsv) in the final output?

"Error executing process > 'NANOPOLISH_VER:getVersion' " when running MOP_TAIL module

Hi devs,

I'm running MOP2 on a SLURM cluster and am running into an issue with the MOP_TAIL module; it fails to pull the image for NanoPolish. As an example, I have run the example yeast data through mop_preprocess and mop_tail with all default settings.

Everything runs fine up until it tries to pull the image for NanoPolish, and gives the following error:

Error executing process > 'NANOPOLISH_VER:getVersion'
Caused by:
  Failed to pull singularity image
  command: singularity pull  --name biocorecrg-mopnanopolish-0.2.img.pulling.1651764466493 docker://biocorecrg/mopnanopolish:0.2 > /dev/null
  status : 255
  message:
    INFO:    Converting OCI blobs to SIF format
    INFO:    Starting build...
    Getting image source signatures
    Copying blob sha256:1e74cf3ea8b17b29912c91cc0ea80b5d3aed0ffe25aa2c8d3d4b24c1454f7052
    ... 
    Copying config sha256:ec012d2e1e8779518bc0840db99d65a9b8ba084662ccbb9f457c09dbb6de27b2
    Writing manifest to image destination
    Storing signatures
    FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: conveyor failed to get: no descriptor found for reference "440b2c8927bbe2d4e64f19fdf5c304bad7fd7f9c70802cb25b07ec77d9243cf0"

I'm guessing it's because it's trying to save the image to /dev/null, which doesn't exist? Not sure if this is a singularity issue or how to go about fixing this...

Attached below are the job output and nextflow logs.

Test_nextflow.log
TailTest-32837074.txt

Edit: I should also mention, it still happens even if I disable NanoPolish in params.config (nano polish : NO).

Thanks,
Sean Robertson

mop_mod Nanocompore eventalign_collapse requires three threads, but uses only one by default. How to adjust tools_opt.tsv?

Hi there,

I am trying to use the mop_mod pipeline with some data that I have already piped into the same output with mop_preprocess. I am using docker as my virtualization software.
However, the nanocompore part of the nextflow pipeline throws up the following error in the eventalign stage (pulled from the log.txt file)

[8f/fcd29c] Submitted process > compore_polish_flow:mean_per_pos (25per_IVT)
[cb/09af45] Submitted process > compore_polish_flow:NANOPOLISH_EVENTALIGN:eventalignCollapse (25per_IVT)
[2f/188da8] Submitted process > compore_polish_flow:NANOPOLISH_EVENTALIGN:eventalignCollapse (no_sample)
ERROR ~ Error executing process > 'compore_polish_flow:NANOPOLISH_EVENTALIGN:eventalignCollapse (25per_IVT)'

Caused by:
Process compore_polish_flow:NANOPOLISH_EVENTALIGN:eventalignCollapse (25per_IVT) terminated with an error exit status (1)

Command executed:

zcat 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_27.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_41.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_16.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_5.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_23.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_19.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_4.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_31.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_28.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_2.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_7.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_34.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_38.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_12.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_13.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_24.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_20.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_8.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_0.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_1.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_42.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_11.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_21.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_6.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_40.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_22.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_35.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_33.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_18.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_39.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_29.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_17.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_37.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_36.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_26.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_32.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_9.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_25.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_15.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_3.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_14.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_10.fast5_event_align.tsv.gz 25per_IVT_AQJ052_pass_ae7f1f60_c0908c38_30.fast5_event_align.tsv.gz | awk '!(/^contig/ && NR>1)' | tee >(pigz -p 1 -9 - > 25per_IVT_combined.eventalign.tsv.gz) | NanopolishComp Eventalign_collapse -t 1 -o 25per_IVT_collapsed_align_events

Command exit status:
1

Command output:
(empty)

Command error:
Checking arguments
Traceback (most recent call last):
File "/usr/local/python/versions/3.6.3/bin/NanopolishComp", line 8, in
sys.exit(main())
File "/usr/local/python/versions/3.6.3/lib/python3.6/site-packages/NanopolishComp/main.py", line 65, in main
args.func(args)
File "/usr/local/python/versions/3.6.3/lib/python3.6/site-packages/NanopolishComp/main.py", line 80, in Eventalign_collapse_main
quiet = args.quiet)
File "/usr/local/python/versions/3.6.3/lib/python3.6/site-packages/NanopolishComp/Eventalign_collapse.py", line 106, in init
raise ValueError ("At least 3 threads required")
ValueError: At least 3 threads required

Work dir:
/home/jon/MOP2/mop_mod/work/cb/09af4521356581ae83a301f25c9762

Tip: you can replicate the issue by changing to the process work dir and entering the command bash .command.run

-- Check '.nextflow.log' file for details
Pipeline BIOCORE@CRG Master of Pore completed!
Started at 2023-12-18T15:49:38.553934Z
Finished at 2023-12-18T16:28:36.529021Z
Time elapsed: 38m 58s
Execution status: failed
WARN: Killing running tasks (2)

It looks like the Eventalign command that's being called has a default thread value (1) that is below it's own required minimum (3).
I have attempted a couple of times to solve this problem using the tools_opt.tsv file in the mop_mod folder, trying "-t 3", "-threads 3", or as below:
#flows tool extrapars
epinano epinano ""
nanocompore nanopolish "NanopolishComp Eventalign_collapse -t 3"
nanocompore nanocompore "--sequence_context 2 --downsample_high_coverage 10000"
tombo_resquiggling tombo ""
tombo_msc tombo ""
tombo_lsc tombo ""

Whilst I am able to run the mop_mod pipeline without nanocompore by turning it off, and produce results for the tombo and epinano modules, I am not able to then pipe these through the mop_consensus pipeline., which quits after only checking and indexing the reference fasta.

However these have not worked.
How might I be able to adjust the parameters or the nextflow pipeline to allow the Eventalign_collapse command to use three threads as required? Will this full directory then be able to be piped into nanoconsensus?

Thanks,

Jon

Error executing testing script

Hi MOP2 team,

I wanted to use the MOP2 pipeline for RNA seq data and ran into some trouble at the testing stage. I hadn't made any changes to the installation file. But the log file says there is a script compilation error.

Here is my command line code:
(base) hp@MacBook-Pro-7 mop_preprocess % nextflow run mop_preprocess.nf -with-docker -bg -profile m1mac > log

Thank you!

Determine Sample with Enriched Modification

Hi, I’m currently working with the Master of Pores 2 to detect RNA modifications. In my analysis, I need to determine which of the two samples being compared has an enrichment of modified RNA at position X.

However, in the final output, I only receive information about differences at the supported k-mer positions. In epinano does indicate the enrichment trend toward a specific sample, but I haven’t found such an indication in nanoconsensus or nanoposish/compore.

Have you encountered this issue before or found a way to extract sample specific enrichment information?

version report for different tools

Hello everyone,
I was wondering whether MOP could also output a plain text file in the /report folder containing all the tool versions used in that run. Would be very helpful for writing manuscripts!
Thanks

mop_mod.nf failed execution

Hi there, I ran the command: nextflow run mop_mod.nf -with-docker > log.txt

But after 6 hours, the execution failed and it also let me know about an error. I'm unsure how to fix this, so I'd appreciate any help! I am currently using the test data parameters.

(...)
executor > local (3)
[0d/41ca87] process > checkRef (Checking yeast_rRNA_ref.fa.gz) [100%] 1 of 1, cached: 1 ✔
[0f/fb5c06] process > epinano_flow:splitReference (Splitting of referenc... [100%] 1 of 1, cached: 1 ✔
[e5/ab50d7] process > epinano_flow:splitBams (Splitting of ko_2_s.bam on... [100%] 6 of 6, cached: 6 ✔
[25/2625ab] process > epinano_flow:indexReference (Indexing pieces00.fa) [100%] 1 of 1, cached: 1 ✔
[e3/d89938] process > epinano_flow:EPINANO_CALC_VAR_FREQUENCIES (wt_2___... [ 33%] 2 of 6, cached: 2
[- ] process > epinano_flow:joinEpinanoRes -
[- ] process > epinano_flow:makeEpinanoPlots_ins -
[- ] process > epinano_flow:makeEpinanoPlots_mis -
[- ] process > epinano_flow:makeEpinanoPlots_del -
[c6/ae2355] process > tombo_common_flow:multiToSingleFast5 (mod___batch_0) [100%] 2 of 2, cached: 2 ✔
[f0/efdd51] process > tombo_common_flow:TOMBO_RESQUIGGLE_RNA:resquiggle_... [100%] 2 of 2, cached: 2 ✔
[c0/aa1c0f] process > getChromInfo (reference.fa) [100%] 1 of 1, cached: 1 ✔
[- ] process > tombo_msc_flow:TOMBO_GET_MODIFICATION_MSC:getModif... [ 0%] 0 of 1
[- ] process > bedGraphToWig_msc -
[- ] process > tombo_lsc_flow:TOMBO_GET_MODIFICATION_LSC:getModif... [ 0%] 0 of 1
[- ] process > bedGraphToWig_lsc -
[- ] process > wigToBigWig -
[- ] process > mergeTomboWigsPlus -
[- ] process > mergeTomboWigsMinus -
[f9/967581] process > EPINANO_VER:getVersion [100%] 1 of 1, cached: 1 ✔
[8f/b21e5f] process > NANOPOLISH_VER:getVersion [100%] 1 of 1, cached: 1 ✔
[8f/8b5c64] process > NANOCOMPORE_VER:getVersion [100%] 1 of 1, cached: 1 ✔
[db/f570b6] process > TOMBO_VER:getVersion [100%] 1 of 1, cached: 1 ✔
Error executing process > 'epinano_flow:EPINANO_CALC_VAR_FREQUENCIES (wt_1___pieces00_s.bam on wt_1)'

Caused by:
Process exceeded running time limit (6h)

Command executed:

Epinano_Variants.py -n 8 -R pieces00.fa -b wt_1___pieces00_s.bam -s $SAM2TSV --type t
for i in *.csv; do gzip $i; done

Command exit status:

Command output:
(empty)

Command error:
wt_1___pieces00_s_TMP_ already exists, will overwrite it
Process Process-2:
Traceback (most recent call last):
File "/usr/local/python/versions/3.6.3/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/local/python/versions/3.6.3/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/project/Epinano1.2.0/Epinano_Variants.py", line 45, in split_tsv_for_per_site_var_freq
firstline = next (tsv)

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.>

Pipeline BIOCORE@CRG Master of Pore completed!
Started at 2022-03-15T17:28:37.791708-04:00
Finished at 2022-03-15T23:28:45.450440-04:00
Time elapsed: 6h 8s
Execution status: failed
WARN: Killing pending tasks (1)

Using pre-trained models for modification prediction (MOP_mod)

Hi all,

I was looking into the MOP_MOD workflows and from what I understand they are implementing Epinano, Nanopolish, Tombo and Nanocompore in KD/KO vs WT modes.

I know Epinano and tombo can use pre-trained models without the KO/KD condition (e.g. Epinano-SMV), I was wondering if there's a way to use that through MOP2 or if you have any plans to implement this in the future?

Preprocess Exits at AssignReads

Work through the pipeline with direct RNA FAST5 files we ran across this error.

Error executing process > 'preprocess_flow:AssignReads (mop_preprocess)'

Caused by:
Process preprocess_flow:AssignReads (mop_preprocess) terminated with an error exit status (1)

Command executed:

samtools view mop_preprocess_anno.bam | awk '{gsub(/XF:Z:/,"",$NF); print $1" "$NF}' |grep -v '__' > mop_preprocess.assigned

Command exit status:
1

Command output:
(empty)

Work dir:
/home/jayradke/MOP2/mop_preprocess/work/77/a48e29182a35faba85e30d1e701103

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named .command.sh

Not sure how to fix this...here is want the parms look like....
conffile. : final_summary_01.txt

fast5 : test.fast5
fastq :

reference : Genome.fa
annotation : Annotation.gtf

granularity. : 1

ref_type : genome
pars_tools : drna_tool_unsplice_guppy6_opt.tsv

output : output

GPU : OFF

basecalling : guppy
demultiplexing : NO
demulti_fast5 : NO

filtering : nanoq
mapping : minimap2

counting : htseq
discovery : NO

cram_conv : YES
subsampling_cram : 50

saveSpace : NO
email :

Any thoughts?

Jay

Starting MoP2 from MoP_TAIL

Hello!

We would like to know if there is any chance we can begin using MoP2 from the MoP_TAIL section. It has been made clear that the input for MoP_TAIL is the output from MoP_PREPROCESS, but we have already basecalled our fast5 files, checked qc and mapped to transcriptome using minimap2.

Given the lengthy time and memory consumption it would take to repeat this step in MoP_PREPROCESS for 6 datasets, we would like to begin from MoP_TAIL. Could there be any issues that may arise from this? Is there a process you would recommend carrying out before we do this? And if you would be so kind, could you share what the output folders of MoP_PREPROCESS would look like so we can simulate this with our already-achieved results to "disguise" it as output from MoP_PREPROCESS?

Thank you so much in advance!

Alex

Error executing process > 'preprocess_simple:NANOCOUNT:nanoCount (null)'

Hi,

I am getting the following error when running mop_preprocess.nf:

WARN: Input tuple does not match input set cardinality declared by process preprocess_simple:NANOCOUNT:nanoCount -- offending value: [FAR90122_pass_d34138fc_317, /Users/j/MOP2/mop_preprocess/work/0c/f2fb769bea201189f938fb7a28f69a/FAR90122_pass_d34138fc_317_s.bam]
Error executing process > 'preprocess_simple:NANOCOUNT:nanoCount (null)'

Caused by:
Not a valid path value type: org.codehaus.groovy.runtime.NullObject (null)

Tip: you can replicate the issue by changing to the process work dir and entering the command bash .command.run

The workflow is completed successfully if I set counting = "NO" in params.config file.

Thanks!
Jesús.

Would like to install in M1chip Mac

I was trying to install MOP2 in a M1-Mac and realized that INSTALL.sh (for installing Guppy) failed. I am very new in using command to do the bioinformatic tool. Does any one can suggest me how to make the installation of Guppy work in mac?

I've tried download the lasted Guppy from Nanopore, move to mop_preprocess/bin, and execute similarly.
cd MOP2/mop_preprocess/bin
ln -s ont-guppy-cpu/bin/guppy_* .
nextflow run mop_preprocess.nf -with-docker -bg -profile m1mac > log

It showed zsh: command not found: nextflow
However, it does not work.

Would like to learn very much!

Andrea

conda singularity related error

Dear MOP2 team,

I am trying to prepocess the test data,
however I got this issue related to guppy.
I didnt found same error in other questions. Do you have an idea of the problem ?
I am running it on my local computer.

Thank you for the support.

N E X T F L O W ~ version 22.04.5
Launching mop_preprocess.nf [special_ekeblad] DSL2 - revision: ec40fe0af4

====================================================
BIOCORE@CRG Master of Pores 2. Preprocessing - N F ~ version 2.0

conffile. : final_summary_01.txt

fast5 : /home/workplace/MoP2/MOP2/mop_preprocess/../data/**/*.fast5
fastq :

reference : /home/workplace/MoP2/MOP2/mop_preprocess/../anno/yeast_rRNA_ref.fa.gz
annotation :

granularity. : 1

ref_type : transcriptome
pars_tools : drna_tool_splice_opt.tsv

output : /home/workplace/MoP2/MOP2/mop_preprocess/output

GPU : OFF

basecalling : guppy
demultiplexing : NO
demulti_fast5 : NO

filtering : nanoq
mapping : graphmap

counting : nanocount
discovery : NO

cram_conv : YES
subsampling_cram : 50

saveSpace : NO
email : [email protected]

Sending the email to [email protected]

----------------------CHECK TOOLS -----------------------------
basecalling : guppy

demultiplexing will be skipped
mapping : graphmap
filtering : nanoq
counting : nanocount
discovery will be skipped

[- ] process > flow1:GUPPY_BASECALL:baseCall -
[- ] process > flow1:NANOQ_FILTER:filter -
[- ] process > preprocess_flow:MinIONQC -
[- ] process > preprocess_flow:RNA2DNA -
[- ] process > preprocess_flow:GRAPHMAP:map -
[- ] process > preprocess_flow:SAMTOOLS_CAT:catAln -
[- ] process > preprocess_flow:SAMTOOLS_SORT:sortAln -
[- ] process > preprocess_flow:SAMTOOLS_INDEX:indexBam -
[- ] process > preprocess_flow:checkRef -
[- ] process > preprocess_flow:bam2Cram -
[- ] process > preprocess_flow:bam2stats -
[- ] process > preprocess_flow:joinAlnStats -
[- ] process > preprocess_flow:NANOPLOT_QC:MOP_nanoPlot -
[- ] process > preprocess_flow:concatenateFastQFiles -
[- ] process > preprocess_flow:FASTQC:fastQC -
[- ] process > preprocess_flow:NANOCOUNT:nanoCount -
[- ] process > preprocess_flow:AssignReads -
[- ] process > preprocess_flow:countStats -
[- ] process > preprocess_flow:joinCountStats -
[- ] process > preprocess_flow:MULTIQC:makeReport -

[- ] process > flow1:GUPPY_BASECALL:baseCall [ 0%] 0 of 2
[- ] process > flow1:NANOQ_FILTER:filter -
[- ] process > preprocess_flow:MinIONQC -
[- ] process > preprocess_flow:RNA2DNA -
[- ] process > preprocess_flow:GRAPHMAP:map -
[- ] process > preprocess_flow:SAMTOOLS_CAT:catAln -
[- ] process > preprocess_flow:SAMTOOLS_SORT:sortAln -
[- ] process > preprocess_flow:SAMTOOLS_INDEX:indexBam -
[- ] process > preprocess_flow:checkRef -
[- ] process > preprocess_flow:bam2Cram -
[- ] process > preprocess_flow:bam2stats -
[- ] process > preprocess_flow:joinAlnStats -
[- ] process > preprocess_flow:NANOPLOT_QC:MOP_nanoPlot -
[- ] process > preprocess_flow:concatenateFastQFiles -
[- ] process > preprocess_flow:FASTQC:fastQC -
[- ] process > preprocess_flow:NANOCOUNT:nanoCount -
[- ] process > preprocess_flow:AssignReads -
[- ] process > preprocess_flow:countStats -
[- ] process > preprocess_flow:joinCountStats -
[- ] process > preprocess_flow:MULTIQC:makeReport -
Pulling Singularity image docker://biocorecrg/mopprepr:0.7 [cache /home/workplace/MoP2/MOP2/mop_preprocess/../singularity/biocorecrg-mopprepr-0.7.img]

executor > local (2)
[21/caa2f7] process > flow1:GUPPY_BASECALL:baseCall (mod---1) [ 0%] 0 of 2
[- ] process > flow1:NANOQ_FILTER:filter -
[- ] process > preprocess_flow:MinIONQC -
[- ] process > preprocess_flow:RNA2DNA -
[- ] process > preprocess_flow:GRAPHMAP:map -
[- ] process > preprocess_flow:SAMTOOLS_CAT:catAln -
[- ] process > preprocess_flow:SAMTOOLS_SORT:sortAln -
[- ] process > preprocess_flow:SAMTOOLS_INDEX:indexBam -
[- ] process > preprocess_flow:checkRef -
[- ] process > preprocess_flow:bam2Cram -
[- ] process > preprocess_flow:bam2stats -
[- ] process > preprocess_flow:joinAlnStats -
[- ] process > preprocess_flow:NANOPLOT_QC:MOP_nanoPlot -
[- ] process > preprocess_flow:concatenateFastQFiles -
[- ] process > preprocess_flow:FASTQC:fastQC -
[- ] process > preprocess_flow:NANOCOUNT:nanoCount -
[- ] process > preprocess_flow:AssignReads -
[- ] process > preprocess_flow:countStats -
[- ] process > preprocess_flow:joinCountStats -
[- ] process > preprocess_flow:MULTIQC:makeReport -
Pulling Singularity image docker://biocorecrg/mopprepr:0.7 [cache /home/workplace/MoP2/MOP2/mop_preprocess/../singularity/biocorecrg-mopprepr-0.7.img]
Error executing process > 'flow1:GUPPY_BASECALL:baseCall (wt---2)'

Caused by:
Process flow1:GUPPY_BASECALL:baseCall (wt---2) terminated with an error exit status (255)

Command executed:

guppy_basecaller --fast5_out --flowcell FLO-MIN106 --kit SQK-RNA002 -i ./ --save_path ./wt---2_out --gpu_runners_per_device 1 --cpu_threads_per_caller 1 --num_callers 1
cat wt---2_out/.fastq >> wt---2.fastq
rm wt---2_out/.fastq
gzip wt---2.fastq

Command exit status:
255

Command output:
(empty)

Command error:
ERROR : Session directory does not exist: /home/workplace/miniconda2/var/singularity/mnt/session
ABORT : Retval = 255

Work dir:
/home/workplace/MoP2/MOP2/mop_preprocess/work/e6/2507fd3eb5346234484be1a085c723

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named .command.sh

Pipeline BIOCORE@CRG Master of Pore - preprocess completed!
Started at 2022-08-18T14:06:01.575654+09:00
Finished at 2022-08-18T14:06:19.136097+09:00
Time elapsed: 17.6s
Execution status: failed
Failed to invoke workflow.onComplete event handler

-- Check script 'mop_preprocess.nf' at line: 632 or see '.nextflow.log' file for more details

executor > local (2)
[21/caa2f7] process > flow1:GUPPY_BASECALL:baseCall (mod---1) [100%] 2 of 2, failed: 2 ✘
[- ] process > flow1:NANOQ_FILTER:filter -
[- ] process > preprocess_flow:MinIONQC -
[- ] process > preprocess_flow:RNA2DNA -
[- ] process > preprocess_flow:GRAPHMAP:map -
[- ] process > preprocess_flow:SAMTOOLS_CAT:catAln -
[- ] process > preprocess_flow:SAMTOOLS_SORT:sortAln -
[- ] process > preprocess_flow:SAMTOOLS_INDEX:indexBam -
[- ] process > preprocess_flow:checkRef -
[- ] process > preprocess_flow:bam2Cram -
[- ] process > preprocess_flow:bam2stats -
[- ] process > preprocess_flow:joinAlnStats -
[- ] process > preprocess_flow:NANOPLOT_QC:MOP_nanoPlot -
[- ] process > preprocess_flow:concatenateFastQFiles -
[- ] process > preprocess_flow:FASTQC:fastQC -
[- ] process > preprocess_flow:NANOCOUNT:nanoCount -
[- ] process > preprocess_flow:AssignReads -
[- ] process > preprocess_flow:countStats -
[- ] process > preprocess_flow:joinCountStats -
[- ] process > preprocess_flow:MULTIQC:makeReport -
Pulling Singularity image docker://biocorecrg/mopprepr:0.7 [cache /home/workplace/MoP2/MOP2/mop_preprocess/../singularity/biocorecrg-mopprepr-0.7.img]
Error executing process > 'flow1:GUPPY_BASECALL:baseCall (wt---2)'

Caused by:
Process flow1:GUPPY_BASECALL:baseCall (wt---2) terminated with an error exit status (255)

Command executed:

Command exit status:
255

Command output:
(empty)

Command error:
ERROR : Session directory does not exist: /home/workplace/miniconda2/var/singularity/mnt/session
ABORT : Retval = 255

Work dir:
/home/workplace/MoP2/MOP2/mop_preprocess/work/e6/2507fd3eb5346234484be1a085c723

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named .command.sh

Preprocess with multiple Fast5 files?

I have gotten the preprocess to work on single file using Docker. However, when I change the command to map to a folder that has multiple Fast5 files I get the following error...

Error executing process > 'flow1:GUPPY_BASECALL:baseCall (mop_preprocess---1)'

Caused by:
Process flow1:GUPPY_BASECALL:baseCall (mop_preprocess---1) terminated with an error exit status (1)

Command executed:

export LD_LIBRARY_PATH="/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs"
guppy_basecaller -x "cuda:0" --fast5_out --flowcell FLO-MIN106 --kit SQK-RNA002 --disable_qscore_filtering -i ./ --save_path ./mop_preprocess---1_out --gpu_runners_per_device 1 --cpu_threads_per_caller 1 --num_callers 1
cat mop_preprocess---1_out/.fastq >> mop_preprocess---1.fastq
rm mop_preprocess---1_out/.fastq
gzip mop_preprocess---1.fastq

Command exit status:
1

Command output:
CRASHPAD MESSAGE:
ONT Guppy basecalling software version 6.3.7+532d626, minimap2 version 2.22-r1101

Do I need to use Singularity instead of Docker for this to work on the GPU with multiple files? Works on GPU with a single file with Docker.

Jay

compore_polish_flow:mean_per_pos terminates with error exit status 1

Hi,

I'm running MOP2 on awsbatch with a relatively minimalistic input setup for testing purposes. I have matching wildtype and invitro DRS data (1 rep each) and i defined a reference for only one gene (which is also covered by the in vitro transcribed data).

While i get some output for epinanoflow. The pipeline terminates with mentioned error. Here's the statement from the log file:

_Error executing process > 'compore_polish_flow:mean_per_pos (invitro)'

Caused by:
Process compore_polish_flow:mean_per_pos (invitro) terminated with an error exit status (1)

Command executed:

mean_per_pos.py -i invitro_AIE407_pass_720e7cf3_16.fast5_event_align.tsv.gz -o basename invitro_AIE407_pass_720e7cf3_16.fast5_event_align.tsv.gz .fast5_event_align.tsv.gz
#gzip *_processed_perpos_median.tsv

Command exit status:
1

Command output:
(empty)

Command error:
thread '' panicked at 'no lines in the file', /github/workspace/polars/polars-io/src/csv_core/parser.rs:108:51
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
Traceback (most recent call last):
File "/project/nextflow-bin/mean_per_pos.py", line 145, in
main()
File "/project/nextflow-bin/mean_per_pos.py", line 127, in main
raw_import = parse_input(a.input)
File "/project/nextflow-bin/mean_per_pos.py", line 34, in parse_input
"event_level_mean": pl.Float32 })
File "/usr/local/python/versions/3.6.3/lib/python3.6/site-packages/polars/io.py", line 405, in read_csv
rechunk=rechunk,
File "/usr/local/python/versions/3.6.3/lib/python3.6/site-packages/polars/internals/frame.py", line 511, in read_csv
parse_dates,
pyo3_runtime.PanicException: no lines in the file_

I've also attatched the command.log and .err files from corresponding work directory, as well as the complete log file. When looking at the previous steps and the fast5_event_align.tsv.gz input file i observed that this file is empty.

Could you give me any clues where the error might stem from? This is the first time I'm using awsbatch and one thing i should mention is that i didn't configured a queue with a gpu-based CE yet, even though the queue is stated in the awsbatch profile.

Any tips probably will help.

Best,
Ben

logs.zip

'Failed to invoke `workflow.onComplete` event handler' in mop_preprocess

So I'm running the mop_preprocess module on a SLURM cluster with the following command:

nextflow run mop_preprocess.nf -with-singularity -profile slurm

It seems to run through without any errors, but fails on the 'workflow.onComplete' function. After running everything successfully and saying 'Execution status : OK', it says:

Failed to invoke 'workflow.onComplete' event handler -- Check script 'mop_preprocess.nf' at line: 631 or see '.nextflow.log' file for more details

mop_preprocess.nf is fine, I didn't touch anything after the parameter setup, and the nextflow.log just says 'java.io.IOException: Broken pipe' once it gets to that part. The full logs are attached below. Any idea how to fix this? Or if it's even a problem?

MOP2_Test_log.txt
nextflow.log

Issue running preprocessing pipeline in MOP2

Hi,

I have been trying unsuccessfully to run the Master of Pores 2, mop_preprocess pipeline. I installed Docker and Guppy as mentioned in the installation doc. My input folder (2xpA-293T) includes several fast5 files. The FASTA reference file is in a separate folder as well as the GTTF annotation file. I defined my parameters in a command line:

./nextflow run /home/lucia/MOP2/mop_preprocess/mop_preprocess.nf -with-docker --fast5_files /home/lucia/MOP_rawdata/2xpA-293T/**/*.fast5 --annotation /home/lucia/annotation/ --reference /home/lucia/reference/ --basecalling guppy --mapping minimap2 --count nanocount --filtering nanofilt --output /home/lucia/MOP_preprocessed/2xpA-293T/ > log.txt

However, the command is not working. I have no error message appearing but I have no output, so I think the command might not be going through all the fast5 files in the input folder. I tried multiple variations of this command line to see if it was a syntax error but the result is always the same. I also reinstalled MOP2. How can I fix this issue?

Here's the log.txt doc:

conffile. : final_summary_01.txt
fast5 : /home/lucia/MOP2/mop_preprocess/../data/**/*.fast5
fastq :
reference : /home/lucia/reference/
annotation : /home/lucia/annotation/
granularity. : 1
ref_type : transcriptome
pars_tools : drna_tool_splice_opt.tsv
output : /home/lucia/MOP_preprocessed/2xpA-293T/
GPU : OFF
basecalling : guppy
demultiplexing : NO
demulti_fast5 : NO
filtering : nanofilt
mapping : minimap2
counting : nanocount
discovery : NO
cram_conv : YES
subsampling_cram : 50
saveSpace : NO
email : [email protected]
/home/outdirs.nf (/home/outdirs.nf)

-- Check script 'MOP2/mop_preprocess/mop_preprocess.nf' at line: 69 or see '.nextflow.log' file for more details

Thank you!

Running mop_preprocess on MacbookPro/macOS Monterey

Hi again! Sorry for all my issue requests, but I am learning bioinformatics as a hobby with RNAseq, therefore, my knowledge on code is very limited!!

So I wanted to run the pipeline on my own time with my laptop, but I am having issues with docker; thus I do not know if I installed Nextflow + Docker + Guppy correctly and I was hoping if anyone had written up a documentation of running MOP2 on an Apple laptop! It would be super helpful to me!

attached is the error I am currently seeing while running mop_preprocess with test configurations!

N E X T F L O W ~ version 21.10.6
Launching mop_preprocess.nf [scruffy_wing] - revision: ecf5447b25

====================================================
BIOCORE@CRG Master of Pores 2. Preprocessing - N F ~ version 2.0

conffile : final_summary_01.txt

fast5 : /Users/anaescobedo/MOP2/mop_preprocess/../data/**/*.fast5
fastq :

reference : /Users/anaescobedo/MOP2/mop_preprocess/../anno/yeast_rRNA_ref.fa.gz
annotation :

granularity : 1

ref_type : transcriptome
pars_tools : drna_tool_splice_opt.tsv

output : /Users/anaescobedo/MOP2/mop_preprocess/output

GPU : OFF

basecalling : guppy
demultiplexing : NO
demulti_fast5 : NO

filtering : nanoq
mapping : graphmap

counting : nanocount
discovery : NO

cram_conv : YES
subsampling_cram : 50

saveSpace : NO

email : [email protected]

Sending the email to [email protected]

----------------------CHECK TOOLS -----------------------------
basecalling : guppy

demultiplexing will be skipped
mapping : graphmap
filtering : nanoq
counting : nanocount
discovery will be skipped

executor > local (1)
[0c/aafd62] process > flow1:GUPPY_BASECALL:baseCall (wt---2) [ 0%] 0 of 2
[- ] process > flow1:NANOQ_FILTER:filter -
[- ] process > preprocess_flow:MinIONQC -
[- ] process > preprocess_flow:RNA2DNA -
[- ] process > preprocess_flow:GRAPHMAP:map -
[- ] process > preprocess_flow:SAMTOOLS_CAT:catAln -
[- ] process > preprocess_flow:SAMTOOLS_SORT:sortAln -
[- ] process > preprocess_flow:SAMTOOLS_INDEX:indexBam -
[- ] process > preprocess_flow:checkRef [ 0%] 0 of 1
[- ] process > preprocess_flow:bam2Cram -
[- ] process > preprocess_flow:bam2stats -
[- ] process > preprocess_flow:joinAlnStats -
[- ] process > preprocess_flow:NANOPLOT_QC:MOP_nanoPlot -
[- ] process > preprocess_flow:concatenateFastQFiles -
[- ] process > preprocess_flow:FASTQC:fastQC -
[- ] process > preprocess_flow:NANOCOUNT:nanoCount -
[- ] process > preprocess_flow:AssignReads -
[- ] process > preprocess_flow:countStats -
[- ] process > preprocess_flow:joinCountStats -
[- ] process > preprocess_flow:MULTIQC:makeReport -

executor > local (2)
[0c/aafd62] process > flow1:GUPPY_BASECALL:baseCall (wt---2) [ 0%] 0 of 2
[- ] process > flow1:NANOQ_FILTER:filter -
[- ] process > preprocess_flow:MinIONQC -
[- ] process > preprocess_flow:RNA2DNA -
[- ] process > preprocess_flow:GRAPHMAP:map -
[- ] process > preprocess_flow:SAMTOOLS_CAT:catAln -
[- ] process > preprocess_flow:SAMTOOLS_SORT:sortAln -
[- ] process > preprocess_flow:SAMTOOLS_INDEX:indexBam -
[e7/f9cd7c] process > preprocess_flow:checkRef (Checking yeast_rRNA_ref.fa.gz) [ 0%] 0 of 1
[- ] process > preprocess_flow:bam2Cram -
[- ] process > preprocess_flow:bam2stats -
[- ] process > preprocess_flow:joinAlnStats -
[- ] process > preprocess_flow:NANOPLOT_QC:MOP_nanoPlot -
[- ] process > preprocess_flow:concatenateFastQFiles -
[- ] process > preprocess_flow:FASTQC:fastQC -
[- ] process > preprocess_flow:NANOCOUNT:nanoCount -
[- ] process > preprocess_flow:AssignReads -
[- ] process > preprocess_flow:countStats -
[- ] process > preprocess_flow:joinCountStats -
[- ] process > preprocess_flow:MULTIQC:makeReport -

Caused by:
Process flow1:GUPPY_BASECALL:baseCall (wt---2) terminated with an error exit status (125)

Command executed:

guppy_basecaller --fast5_out --flowcell FLO-MIN106 --kit SQK-RNA002 -i ./ --save_path ./wt---2_out --gpu_runners_per_device 1 --cpu_threads_per_caller 1 --num_callers 8
cat wt---2_out/.fastq >> wt---2.fastq
rm wt---2_out/.fastq
gzip wt---2.fastq

Command exit status:
125

Command output:
(empty)

Command error:
docker: Error response from daemon: Range of CPUs is from 0.01 to 4.00, as there are only 4 CPUs available.
See 'docker run --help'.

Work dir:
/Users/anaescobedo/MOP2/mop_preprocess/work/0c/aafd6231abadda45a3e00e0446f905

Tip: you can replicate the issue by changing to the process work dir and entering the command bash .command.run

Pipeline BIOCORE@CRG Master of Pore - preprocess completed!
Started at 2022-04-06T17:17:10.093-04:00
Finished at 2022-04-06T17:17:24.933-04:00
Time elapsed: 14.8s
Execution status: failed

Guppy version

Hi, Luca,

I installed MOP2 and installed Guppy 6.4.6. Then it showed : unlink: cannot unlink ‘mop_preprocess/bin/ont-guppy/lib/libz.so’: No such file or directory

And, I changed pars_tools to drna_tool_unsplice_guppy6_opt.tsv.

Then I ran nextflow run mop_preprocess.nf -with-singularity

Error msg was received as

N E X T F L O W  ~  version 22.04.0
Launching `mop_preprocess.nf` [naughty_babbage] DSL2 - revision: ec40fe0af4


╔╦╗╔═╗╔═╗  ╔═╗┬─┐┌─┐┌─┐┬─┐┌─┐┌─┐┌─┐┌─┐┌─┐
║║║║ ║╠═╝  ╠═╝├┬┘├┤ ├─┘├┬┘│ ││  ├┤ └─┐└─┐
╩ ╩╚═╝╩    ╩  ┴└─└─┘┴  ┴└─└─┘└─┘└─┘└─┘└─┘
                                                                                       
====================================================
BIOCORE@CRG Master of Pores 2. Preprocessing - N F  ~  version 2.0
====================================================

conffile.                 : final_summary_01.txt

fast5                     : /staging/biology/andreachi77/MOP2/mop_preprocess/../data/**/*.fast5
fastq                     : 

reference                 : /staging/biology/andreachi77/MOP2/mop_preprocess/../anno/yeast_rRNA_ref.fa.gz
annotation                : 

granularity.              : 1

ref_type                  : transcriptome
pars_tools                : drna_tool_unsplice_guppy6_opt.tsv

output                    : /staging/biology/andreachi77/MOP2/mop_preprocess/output

GPU                       : OFF

basecalling               : guppy 
demultiplexing            : NO 
demulti_fast5             : NO

filtering                 : nanoq
mapping                   : graphmap

counting                  : nanocount
discovery                 : NO

cram_conv           	  : YES
subsampling_cram          : 50


saveSpace                 : NO
email                     : [email protected]

Sending the email to [email protected]

----------------------CHECK TOOLS -----------------------------
basecalling : guppy
> demultiplexing will be skipped
mapping : graphmap
filtering : nanoq
counting : nanocount
> discovery will be skipped
--------------------------------------------------------------
[-        ] process > flow1:GUPPY_BASECALL:baseCall            -
[-        ] process > flow1:NANOQ_FILTER:filter                -
[-        ] process > preprocess_flow:MinIONQC                 -
[-        ] process > preprocess_flow:RNA2DNA                  -
[-        ] process > preprocess_flow:GRAPHMAP:map             -
[-        ] process > preprocess_flow:SAMTOOLS_CAT:catAln      -
[-        ] process > preprocess_flow:SAMTOOLS_SORT:sortAln    -
[-        ] process > preprocess_flow:SAMTOOLS_INDEX:indexBam  -
[-        ] process > preprocess_flow:checkRef                 -
[-        ] process > preprocess_flow:bam2Cram                 -
[-        ] process > preprocess_flow:bam2stats                -
[-        ] process > preprocess_flow:joinAlnStats             -
[-        ] process > preprocess_flow:NANOPLOT_QC:MOP_nanoPlot -
[-        ] process > preprocess_flow:concatenateFastQFiles    -
[-        ] process > preprocess_flow:FASTQC:fastQC            -

[-        ] process > flow1:GUPPY_BASECALL:baseCall            [  0%] 0 of 2
[-        ] process > flow1:NANOQ_FILTER:filter                -
[-        ] process > preprocess_flow:MinIONQC                 -
[-        ] process > preprocess_flow:RNA2DNA                  -
[-        ] process > preprocess_flow:GRAPHMAP:map             -
[-        ] process > preprocess_flow:SAMTOOLS_CAT:catAln      -
[-        ] process > preprocess_flow:SAMTOOLS_SORT:sortAln    -
[-        ] process > preprocess_flow:SAMTOOLS_INDEX:indexBam  -
[-        ] process > preprocess_flow:checkRef                 -
[-        ] process > preprocess_flow:bam2Cram                 -
[-        ] process > preprocess_flow:bam2stats                -
[-        ] process > preprocess_flow:joinAlnStats             -
[-        ] process > preprocess_flow:NANOPLOT_QC:MOP_nanoPlot -
[-        ] process > preprocess_flow:concatenateFastQFiles    -
[-        ] process > preprocess_flow:FASTQC:fastQC            -
[-        ] process > preprocess_flow:NANOCOUNT:nanoCount      -
[-        ] process > preprocess_flow:AssignReads              -
[-        ] process > preprocess_flow:countStats               -
[-        ] process > preprocess_flow:joinCountStats           -
[-        ] process > preprocess_flow:MULTIQC:makeReport       -
Pulling Singularity image docker://biocorecrg/mopprepr:0.7 [cache /staging/biology/andreachi77/MOP2/mop_preprocess/../singularity/biocorecrg-mopprepr-0.7.img]

executor >  local (2)
[8e/a72fcf] process > flow1:GUPPY_BASECALL:baseCall (mod---2)  [  0%] 0 of 2
[-        ] process > flow1:NANOQ_FILTER:filter                -
[-        ] process > preprocess_flow:MinIONQC                 -
[-        ] process > preprocess_flow:RNA2DNA                  -
[-        ] process > preprocess_flow:GRAPHMAP:map             -
[-        ] process > preprocess_flow:SAMTOOLS_CAT:catAln      -
[-        ] process > preprocess_flow:SAMTOOLS_SORT:sortAln    -
[-        ] process > preprocess_flow:SAMTOOLS_INDEX:indexBam  -
[-        ] process > preprocess_flow:checkRef                 -
[-        ] process > preprocess_flow:bam2Cram                 -
[-        ] process > preprocess_flow:bam2stats                -
[-        ] process > preprocess_flow:joinAlnStats             -
[-        ] process > preprocess_flow:NANOPLOT_QC:MOP_nanoPlot -
[-        ] process > preprocess_flow:concatenateFastQFiles    -
[-        ] process > preprocess_flow:FASTQC:fastQC            -
[-        ] process > preprocess_flow:NANOCOUNT:nanoCount      -
[-        ] process > preprocess_flow:AssignReads              -
[-        ] process > preprocess_flow:countStats               -
[-        ] process > preprocess_flow:joinCountStats           -
[-        ] process > preprocess_flow:MULTIQC:makeReport       -
Pulling Singularity image docker://biocorecrg/mopprepr:0.7 [cache /staging/biology/andreachi77/MOP2/mop_preprocess/../singularity/biocorecrg-mopprepr-0.7.img]

executor >  local (2)
[8e/a72fcf] process > flow1:GUPPY_BASECALL:baseCall (mod---2)  [  0%] 0 of 2
[-        ] process > flow1:NANOQ_FILTER:filter                -
[-        ] process > preprocess_flow:MinIONQC                 -
[-        ] process > preprocess_flow:RNA2DNA                  -
[-        ] process > preprocess_flow:GRAPHMAP:map             -
[-        ] process > preprocess_flow:SAMTOOLS_CAT:catAln      -
[-        ] process > preprocess_flow:SAMTOOLS_SORT:sortAln    -
[-        ] process > preprocess_flow:SAMTOOLS_INDEX:indexBam  -
[-        ] process > preprocess_flow:checkRef                 -
[-        ] process > preprocess_flow:bam2Cram                 -
[-        ] process > preprocess_flow:bam2stats                -
[-        ] process > preprocess_flow:joinAlnStats             -
[-        ] process > preprocess_flow:NANOPLOT_QC:MOP_nanoPlot -
[-        ] process > preprocess_flow:concatenateFastQFiles    -
[-        ] process > preprocess_flow:FASTQC:fastQC            -
[-        ] process > preprocess_flow:NANOCOUNT:nanoCount      -
[-        ] process > preprocess_flow:AssignReads              -
[-        ] process > preprocess_flow:countStats               -
[-        ] process > preprocess_flow:joinCountStats           -
[-        ] process > preprocess_flow:MULTIQC:makeReport       -
Pulling Singularity image docker://biocorecrg/mopprepr:0.7 [cache /staging/biology/andreachi77/MOP2/mop_preprocess/../singularity/biocorecrg-mopprepr-0.7.img]
Error executing process > 'flow1:GUPPY_BASECALL:baseCall (wt---1)'

Caused by:
  Process `flow1:GUPPY_BASECALL:baseCall (wt---1)` terminated with an error exit status (255)

Command executed:

  guppy_basecaller          --fast5_out --flowcell FLO-MIN106 --kit SQK-RNA002 --disable_qscore_filtering -i ./         --save_path ./wt---1_out         --gpu_runners_per_device 1         --cpu_threads_per_caller 1 	    --num_callers  8
  cat wt---1_out/*.fastq >> wt---1.fastq
  rm wt---1_out/*.fastq
  gzip wt---1.fastq

Command exit status:
  255

Command output:
  (empty)

Command error:
  Unexpected token '--fast5_out' on command-line.
  Try 'guppy_basecaller --help' for more information.

Work dir:
  /staging/biology/andreachi77/MOP2/mop_preprocess/work/c7/67d1a19645911d99ddce1032319373

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`


Pipeline BIOCORE@CRG Master of Pore - preprocess completed!
Started at  2023-04-13T15:07:57.935074+08:00
Finished at 2023-04-13T15:08:07.148731+08:00
Time elapsed: 9.2s
Execution status: failed
Failed to invoke `workflow.onComplete` event handler

 -- Check script 'mop_preprocess.nf' at line: 632 or see '.nextflow.log' file for more details

executor >  local (2)
[8e/a72fcf] process > flow1:GUPPY_BASECALL:baseCall (mod---2)  [100%] 2 of 2, failed: 2 ✘
[-        ] process > flow1:NANOQ_FILTER:filter                -
[-        ] process > preprocess_flow:MinIONQC                 -
[-        ] process > preprocess_flow:RNA2DNA                  -
[-        ] process > preprocess_flow:GRAPHMAP:map             -
[-        ] process > preprocess_flow:SAMTOOLS_CAT:catAln      -
[-        ] process > preprocess_flow:SAMTOOLS_SORT:sortAln    -
[-        ] process > preprocess_flow:SAMTOOLS_INDEX:indexBam  -
[-        ] process > preprocess_flow:checkRef                 -
[-        ] process > preprocess_flow:bam2Cram                 -
[-        ] process > preprocess_flow:bam2stats                -
[-        ] process > preprocess_flow:joinAlnStats             -
[-        ] process > preprocess_flow:NANOPLOT_QC:MOP_nanoPlot -
[-        ] process > preprocess_flow:concatenateFastQFiles    -
[-        ] process > preprocess_flow:FASTQC:fastQC            -
[-        ] process > preprocess_flow:NANOCOUNT:nanoCount      -
[-        ] process > preprocess_flow:AssignReads              -
[-        ] process > preprocess_flow:countStats               -
[-        ] process > preprocess_flow:joinCountStats           -
[-        ] process > preprocess_flow:MULTIQC:makeReport       -
Pulling Singularity image docker://biocorecrg/mopprepr:0.7 [cache /staging/biology/andreachi77/MOP2/mop_preprocess/../singularity/biocorecrg-mopprepr-0.7.img]
Error executing process > 'flow1:GUPPY_BASECALL:baseCall (wt---1)'

Caused by:
  Process `flow1:GUPPY_BASECALL:baseCall (wt---1)` terminated with an error exit status (255)

Command executed:

  guppy_basecaller          --fast5_out --flowcell FLO-MIN106 --kit SQK-RNA002 --disable_qscore_filtering -i ./         --save_path ./wt---1_out         --gpu_runners_per_device 1         --cpu_threads_per_caller 1 	    --num_callers  8
  cat wt---1_out/*.fastq >> wt---1.fastq
  rm wt---1_out/*.fastq
  gzip wt---1.fastq

Command exit status:
  255

Command output:
  (empty)

Command error:
  Unexpected token '--fast5_out' on command-line.
  Try 'guppy_basecaller --help' for more information.

Work dir:
  /staging/biology/andreachi77/MOP2/mop_preprocess/work/c7/67d1a19645911d99ddce1032319373

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

Is it better just to use older version Guppy?

Thanks!

Docker issue

Hi,

Running install as per docs and the first test on an EC2 instance , but with docker and i get an issue:

docker: failed to register layer: ApplyLayer exit status 1 stdout: stderr: lchown /project/minimap2-2.17_x64-linux: invalid argument
I am not sure what the issue is ?

cant find fastq file

Hi, first time running this tool. Trying to run "Testing" part. However it seems like the cat command can't find the fastq files created by guppy.

Pulling Singularity image docker://biocorecrg/mopbasecall:0.2 [cache /gstore/home/.../repos/MOP2/mop_preprocess/../singularity/biocorecrg-mopbasecall-0.2.img]
Pulling Singularity image docker://biocorecrg/mopprepr:0.7 [cache /gstore/home/.../repos/MOP2/mop_preprocess/../singularity/biocorecrg-mopprepr-0.7.img]
[c4/c48b78] Submitted process > flow1:GUPPY_BASECALL:baseCall (mod---2)
[75/1a5a50] Submitted process > flow1:GUPPY_BASECALL:baseCall (wt---1)
[0d/153cf4] Submitted process > preprocess_flow:checkRef (Checking yeast_rRNA_ref.fa.gz)
Error executing process > 'flow1:GUPPY_BASECALL:baseCall (wt---1)'

Caused by:
  Process `flow1:GUPPY_BASECALL:baseCall (wt---1)` terminated with an error exit status (1)

Command executed:

  guppy_basecaller          --fast5_out --flowcell FLO-MIN106 --kit SQK-RNA002  -i ./         --save_path ./wt---1_out         --gpu_runners_per_device 1         --cpu_threads_per_caller 1 	    --num_callers  1
  cat wt---1_out/*.fastq >> wt---1.fastq
  rm wt---1_out/*.fastq
  gzip wt---1.fastq

Command exit status:
  1

Command output:
  ONT Guppy basecalling software version 6.0.1+652ffd1
  config file:        /gstore/home/.../repos/MOP2/mop_preprocess/bin/ont-guppy/data/rna_r9.4.1_70bps_hac.cfg
  model file:         /gstore/home/.../repos/MOP2/mop_preprocess/bin/ont-guppy/data/template_rna_r9.4.1_70bps_hac.jsn
  input path:         ./
  save path:          ./wt---1_out
  chunk size:         2000
  chunks per runner:  512
  minimum qscore:     7
  records per file:   4000
  num basecallers:    1
  cpu mode:           ON
  threads per caller: 1
  Found 1 fast5 files to process.
  Init time: 166 ms
  
  0%   10   20   30   40   50   60   70   80   90   100%
  |----|----|----|----|----|----|----|----|----|----|
  ***************************************************
  Caller time: 1044688 ms, Samples called: 28004665, samples/s: 26806.7
  Finishing up any open output files.
  Basecalling completed successfully.

Command error:
  cat: 'wt---1_out/*.fastq': No such file or directory

Work dir:
  /gstore/scratch/u/.../nxf_work/75/1a5a50e2358b02eff8db4e6e85b584

Browsing to that wt---1 directory I see that fastq files are one more level deeper under that "pass" directory. Perhaps that's the issue? Here is the directory listing:

[wt---1_out]ls -R
.:
fail  guppy_basecaller_log-2022-03-12_11-51-58.log  pass  sequencing_summary.txt  sequencing_telemetry.js  workspace

./fail:
fastq_runid_61e803f07f123a6fc6f41fc5122323d9326781a1_0_0.fastq

./pass:
fastq_runid_61e803f07f123a6fc6f41fc5122323d9326781a1_0_0.fastq

./workspace:
batch_0.fast5

MOP preprocess issue: no descriptor found for reference

Hi, I was running MOP preprocess on our cluster with the following setup:

nextflow run mop_preprocess.nf -with-singularity -bg -profile slurm > log.txt

It produced fastq, bam and fast5_final_summary.stats files but nothing else.

This is how the error looked, I'm also attaching the log file:

Error executing process > 'preprocess_flow:NANOPLOT_QC:MOP_nanoPlot (fast5)'

Caused by:
  Failed to pull singularity image
  command: singularity pull  --name biocorecrg-nanoplot-1.33.0.img.pulling.1690999883608 docker://biocorecrg/nanoplot:1.33.0 > /dev/null
  status : 255
  message:
    INFO:    Converting OCI blobs to SIF format
    INFO:    Starting build...
    Getting image source signatures
    Copying blob sha256:e518f22d2fe88cb8e3a813783655e7615b90d288ebd5e683e43b14c7769f69de
    Copying blob sha256:1e74cf3ea8b17b29912c91cc0ea80b5d3aed0ffe25aa2c8d3d4b24c1454f7052
    Copying blob sha256:e9092151f7a2491c0d2b42f59c1b8e2bf859ac6e53321eba7bb1017b2b3fe43f
    Copying blob sha256:276898f99f7b162c4834f6e13978f98bfd5f377635cdc1416d1fe27d38b47820
    Copying blob sha256:a521508fa11e33da609b8f0480253ef148b68b8beb56c0e409d0d207fda5a363
    Copying blob sha256:61af738b4846d9ef2e9eea025fb44288da5b8c76e8289a4f9eb62a2beb58626e
    Copying blob sha256:a808f8bcd824acac4d82f9468a5b093eddc46a12468f99bab3c6191e9dd429ac
    Copying blob sha256:7f4f7d6e8dc6d3bc92ecb3f182a4177c46653fcf4a4e3eaecdebdebd6eebcf5d
    Copying blob sha256:3762fddd8cf345cbdd4413f62bf112b2a194c330803a1290e4582c26a357a2a6
    Copying blob sha256:f5624c74d9d1c71e06ad4ca643f473e837b6696d5790a1c8c6c34fdbd1c5a52d
    Copying blob sha256:047916eec6087fc7002822733b41d8b4f76e06cbfee36edee420f8e19730f858
    Copying blob sha256:b0c85be337e40513d2c7cdd5760f24f66aa9f4e5b092f8de6ee10fbdd8c2a3bf
    Copying blob sha256:3209466925d146d6b418fd61720b75b3230e41362b02a66b10aed61ffb5224d8
    Copying blob sha256:773a9d1e092db6b45a50d8cb5873399c5d306351076b0bafd8928e9d9ae64c21
    Copying blob sha256:2a7e8e5f27b96fe53b4ca20a1b926539726f0372feaa314a5d01b2cc5bdcf012
    Copying blob sha256:04920849e9bb4b8e84737fea0e1c9f488c009065a557214907f105bce2b9b5f7
    Copying blob sha256:6a4fbb1ca9cfd43f9964c976f740874ac7bd2ab74fd79b06dea02544fba63643
    Copying blob sha256:4b1ed5a54e490ae7b798d8e75b4736066b7de4ff8a657a1d26f4a37bbdf7dc61
    Copying blob sha256:60f68c6a8169ac7a058cdca8afe87f2725b983095f1dcfe91750322c15026a0e
    Copying blob sha256:dc77bc6ebd0d75399abf2989daa3ff972f9181e009e43280e4ce30da158006ba
    Copying blob sha256:f57ce47e580ed88d218150a1d87b3a039cd69ce3501862d469425eb92a34004e
    Copying blob sha256:066dec84918bd194e52542a07d4b1d6207fec6015523e24e15298784f7548fb7
    Copying blob sha256:a4a559aaec8e982a713b25e2cc6f7eb9b8ab5f4232b8a9c28ac2e93904bee291
    Copying blob sha256:b17a80a838fb01ec770318f8afcba7ebce9d58bc46e0aa72bff68dbb74760b82
    Copying blob sha256:fe3438c68c1da5b4b76642833dcefb6b9f7a16c19fb99ff940c7e1cf997a7fca
    Copying blob sha256:085e113f5f7b22814079c7bb7a8c64c2b723dd08d8eec605ed76b3a5bf4b68b2
    Copying blob sha256:2ee85af3b452dde4b854fec447418c5c84a39ac02a226def808f9054e5d76f7c
    Copying blob sha256:da66a89a03a777217eaf85af8ee22660d6d4e7c0949b6f9ff44a88f4396f6dfb
    Copying blob sha256:3ac11ece5aab976dafa438e4a00e3f807be1232eb56165155a6b7d29568cc9a3
    Copying blob sha256:1394acde2a81ca47392fa5f9a7b043963cd2880a8f87b83ba6ccf126e1252fca
    Copying blob sha256:e38f5712d40c6cf83d77cbf1406f91c6b0d963e83230070b46ff583bf83fd305
    Copying blob sha256:28484ff34ae6bc18658b89c09c4b931f66d14291fb4d7f85063ff0831f5e91e5
    Copying blob sha256:3d6abfb6e258a862708341abf0b0e09a9d9c0caeccb910ce768accfd7dcfa3b3
    Copying blob sha256:ec1e8c1bca9cc3509237a9e5fce03dfc37d72e5effa4a2b39f047dcbf4615951
    Copying blob sha256:dda8c94cbe1d754c0067bdafe8aea99206ed98da5cdb4b05f189921575f8336b
    Copying config sha256:3ed2c2ad5e962b619397efb69caead8225b869e49c65fd42bea2bc40b6fc6dfa
    Writing manifest to image destination
    Storing signatures
    FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: conveyor failed to get: no descriptor found for reference "sha256.afe9f3e2aa92e0302cc1f10033c7b7e978ffd478d4bc8b1a3915261c176c8708"

Any advice on how to go about this would be appreciated, thanks!

Log file here:
log.txt

Issue Running MOP2

Hi there, I have followed your steps to run MOP but I cannot seem to have any success. Could you help me out by letting me know if there are any errors in my configurations at the Preprocess step?

N E X T F L O W ~ version 21.04.3
Launching mop_preprocess.nf [determined_carson] - revision: ecf5447b25

====================================================
BIOCORE@CRG Master of Pores 2. Preprocessing - N F ~ version 2.0

conffile : final_summary_01.txt

fast5 : /home/cart/EpiTX/master_of_pores/NanoPreprocess/data/41147_pellet/pellet/*.fast5
fastq :

reference : /home/cart/EpiTX/MOP2/Reference /GRCh38.primary_assembly.genome.fa.gz
annotation : /home/cart/EpiTX/MOP2/Reference/gencode.v39.annotation.gtf.gz

granularity : 1

ref_type : genome
pars_tools : drna_tool_splice_opt.tsv

output :

GPU : OFF

basecalling : guppy
demultiplexing : guppy
demulti_fast5 : NO

filtering : nanofilt
mapping : graphmap2

counting : nanocount
discovery : NO

cram_conv : YES
subsampling_cram : 50

saveSpace : NO

email :

----------------------CHECK TOOLS -----------------------------
basecalling : guppy
demultiplexing : guppy
mapping : graphmap2
filtering : nanofilt
counting : nanocount

discovery will be skipped

[- ] process > flow2:GUPPY_BASECALL_DEMULTI:baseCallAndDemultiPlex -
[- ] process > flow2:NANOFILT_FILTER:filter -
[- ] process > preprocess_flow:MinIONQC -
[- ] process > preprocess_flow:GRAPHMAP2:map -
[- ] process > preprocess_flow:SAMTOOLS_CAT:catAln -
[- ] process > preprocess_flow:SAMTOOLS_SORT:sortAln -
[- ] process > preprocess_flow:SAMTOOLS_INDEX:indexBam -
[- ] process > preprocess_flow:checkRef

Epinano workflow, how to get per-read results

I have a question about the epinano workflow. As I understand it, the code uses Epinano_variants.py script to compute reference misalignment frequencies at positions in order to compute per-site statistics.
There is a second step in Epinano library (implemented in Epinano_differr.R) which computes different statistics including per_read outliers and generates some plots. MOP2 doesn't use this script, am I correct?
Is there any replacement for this second step in MOP2? Was this step removed/not implemented on purpose?
Thank you for your work in this useful package. :)

Issues with install/testing?

Hi,

Thank you so much for making this interesting tool available! Today I followed the documentation on how to install and everything seemed to go well. When I tried running the command you report under 'Testing', I got something weird.

command:

nextflow run mop_preprocess.nf -with-singularity -bg -profile local > log

contents of log file:

N E X T F L O W  ~  version 18.10.1
Launching `mop_preprocess.nf` [desperate_church] - revision: ecf5447b25
ERROR ~ unable to resolve class GUPPY_BASECALL
 @ line 152, column 26.
   include { GET_WORKFLOWS; BASECALL as GUPPY_BASECALL; BASECALL_DEMULTI as GUPPY_BASECALL_DEMULTI } from "${subworkflowsDir}/basecalling/guppy" addParams(EXTRAPARS_BC: guppy_basecall_pars, EXTRAPARS_DEM: progPars["demultiplexing--guppy"], LABEL: guppy_basecall_label, GPU_OPTION: gpu, MOP: "YES", OUTPUT: output_bc, OUTPUTMODE: outmode)
                            ^

_nf_script_3f0a42d3: 152: unable to resolve class GUPPY_BASECALL_DEMULTI
 @ line 152, column 54.
   ; BASECALL as GUPPY_BASECALL; BASECALL_D
                                 ^

_nf_script_3f0a42d3: 153: unable to resolve class DEMULTIPLEX_VER
 @ line 153, column 11.
   include { GET_VERSION as DEMULTIPLEX_VER; DEMULTIPLEX as DEMULTIPLEX_DEEPLEXICON } from "${subworkflowsDir}/demultiplexing/deeplexicon" addParams(EXTRAPARS: progPars["demultiplexing--deeplexicon"], LABEL:deeplexi_basecall_label, GPU_OPTION: gpu)
             ^

..and it keeps going and reports a total of 37 similar errors. I'll attach the whole log file.

log.txt

What's not going well? Is it me (I have zero nextflow experience)?

Thanks,

Francesco

Error executing process > 'compore_polish_flow:NANOPOLISH_EVENTALIGN:index (mod)'

Good day, colleagues,

Have an error.

ubuntu@ubuntu:~/MOP2/mop_mod$ /home/ubuntu/nextflow run /home/ubuntu/MOP2/mop_mod/mop_mod.nf -with-singularity -profile local
N E X T F L O W  ~  version 22.10.2
Launching `/home/ubuntu/MOP2/mop_mod/mop_mod.nf` [ecstatic_feynman] DSL2 - revision: 83320fa996


╔╦╗╔═╗╔═╗  ╔╦╗┌─┐┌┬┐
║║║║ ║╠═╝  ║║║│ │ ││
╩ ╩╚═╝╩    ╩ ╩└─┘─┴┘
                                                                                       
====================================================
BIOCORE@CRG Master of Pores 2. Detection of RNA modification - N F  ~  version 2.0
====================================================

*****************   Input files    *******************
input_path                              : /home/ubuntu/MOP2/mop_preprocess/output
comparison                              : /home/ubuntu/MOP2/mop_mod/comparison.tsv

********** reference has to be the genome *************
reference                               : /home/ubuntu/MOP2/data/GRCh38_latest_rna.fna.gz
output                                  : /home/ubuntu/MOP2/mop_mod/output

pars_tools				: /home/ubuntu/MOP2/mop_mod/tools_opt.tsv

************************* Flows *******************************
epinano                             	: YES
nanocompore                             : YES
tombo_lsc                               : YES
tombo_msc                               : YES

email                                   : 

Skipping the email

executor >  local (1)
[-        ] process > checkRef                                            [  0%] 0 of 1
[-        ] process > epinano_flow:splitReference                         -
[-        ] process > epinano_flow:splitBams                              -
executor >  local (2)
[ee/87a03a] process > checkRef (Checking GRCh38_latest_rna.fna.gz)        [  0%] 0 of 1
[-        ] process > epinano_flow:splitReference                         -
[-        ] process > epinano_flow:splitBams                              -
executor >  local (2)
[ee/87a03a] process > checkRef (Checking GRCh38_latest_rna.fna.gz)        [  0%] 0 of 1
[-        ] process > epinano_flow:splitReference                         -
[-        ] process > epinano_flow:splitBams                              -
executor >  local (3)
[ee/87a03a] process > checkRef (Checking GRCh38_latest_rna.fna.gz)        [100%] 1 of 1 ✔
[-        ] process > epinano_flow:splitReference                         -
[-        ] process > epinano_flow:splitBams                              -
[-        ] process > epinano_flow:indexReference                         -
executor >  local (3)
[ee/87a03a] process > checkRef (Checking GRCh38_latest_rna.fna.gz)        [100%] 1 of 1 ✔
[-        ] process > epinano_flow:splitReference                         -
[-        ] process > epinano_flow:splitBams                              -
[-        ] process > epinano_flow:indexReference                         -
executor >  local (4)
[ee/87a03a] process > checkRef (Checking GRCh38_latest_rna.fna.gz)        [100%] 1 of 1 ✔
[-        ] process > epinano_flow:splitReference                         -
[-        ] process > epinano_flow:splitBams                              -
[-        ] process > epinano_flow:indexReference                         -
executor >  local (4)
[ee/87a03a] process > checkRef (Checking GRCh38_latest_rna.fna.gz)        [100%] 1 of 1 ✔
[-        ] process > epinano_flow:splitReference                         -
[-        ] process > epinano_flow:splitBams                              -
[-        ] process > epinano_flow:indexReference                         -
executor >  local (4)
[ee/87a03a] process > checkRef (Checking GRCh38_latest_rna.fna.gz)        [100%] 1 of 1 ✔
[-        ] process > epinano_flow:splitReference                         -
[-        ] process > epinano_flow:splitBams                              -
[-        ] process > epinano_flow:indexReference                         -
executor >  local (4)
[ee/87a03a] process > checkRef (Checking GRCh38_latest_rna.fna.gz)        [100%] 1 of 1 ✔
[-        ] process > epinano_flow:splitReference                         -
[-        ] process > epinano_flow:splitBams                              -
[-        ] process > epinano_flow:indexReference                         -
[-        ] process > epinano_flow:EPINANO_CALC_VAR_FREQUENCIES           -
[-        ] process > epinano_flow:joinEpinanoRes                         -
[-        ] process > epinano_flow:makeEpinanoPlots_ins                   -
[-        ] process > epinano_flow:makeEpinanoPlots_mis                   -
[-        ] process > epinano_flow:makeEpinanoPlots_del                   -
[1f/508eec] process > compore_polish_flow:getChromInfo (reference.fa)     [100%] 1 of 1 ✔
[-        ] process > compore_polish_flow:NANOPOLISH_EVENTALIGN:index     -
[-        ] process > compore_polish_flow:NANOPOLISH_EVENTALIGN:eventa... -
[-        ] process > compore_polish_flow:NANOPOLISH_EVENTALIGN:eventa... -
[-        ] process > compore_polish_flow:mean_per_pos                    -
[-        ] process > compore_polish_flow:concat_mean_per_pos             -
[-        ] process > compore_polish_flow:concat_csv_files                -
[-        ] process > compore_polish_flow:NANOCOMPORE_SAMPLE_COMPARE:s... -
[-        ] process > tombo_common_flow:multiToSingleFast5                -
[-        ] process > tombo_common_flow:TOMBO_RESQUIGGLE_RNA:resquiggl... -
[d7/1632d4] process > getChromInfo (reference.fa)                         [100%] 1 of 1 ✔
[-        ] process > tombo_msc_flow:TOMBO_GET_MODIFICATION_MSC:getMod... -
[-        ] process > bedGraphToWig_msc                                   -
[-        ] process > tombo_lsc_flow:TOMBO_GET_MODIFICATION_LSC:getMod... -
[-        ] process > bedGraphToWig_lsc                                   -
[-        ] process > wigToBigWig                                         -
[-        ] process > mergeTomboWigsPlus                                  -
[-        ] process > mergeTomboWigsMinus                                 -
[-        ] process > EPINANO_VER:getVersion                              -
[-        ] process > NANOPOLISH_VER:getVersion                           -
[29/5e20d5] process > NANOCOMPORE_VER:getVersion                          [100%] 1 of 1 ✔
[-        ] process > TOMBO_VER:getVersion                                -





Error executing process > 'compore_polish_flow:NANOPOLISH_EVENTALIGN:index (mod)'

Caused by:
  Failed to pull singularity image
  command: singularity pull  --name biocorecrg-mopnanopolish-0.2.img.pulling.1669106082892 docker://biocorecrg/mopnanopolish:0.2 > /dev/null
  status : 255
  message:
    INFO:    Converting OCI blobs to SIF format
    INFO:    Starting build...
    Getting image source signatures
    Copying blob sha256:1e74cf3ea8b17b29912c91cc0ea80b5d3aed0ffe25aa2c8d3d4b24c1454f7052
    Copying blob sha256:e9092151f7a2491c0d2b42f59c1b8e2bf859ac6e53321eba7bb1017b2b3fe43f
    Copying blob sha256:276898f99f7b162c4834f6e13978f98bfd5f377635cdc1416d1fe27d38b47820
    Copying blob sha256:a521508fa11e33da609b8f0480253ef148b68b8beb56c0e409d0d207fda5a363
    Copying blob sha256:61af738b4846d9ef2e9eea025fb44288da5b8c76e8289a4f9eb62a2beb58626e
    Copying blob sha256:e518f22d2fe88cb8e3a813783655e7615b90d288ebd5e683e43b14c7769f69de
    Copying blob sha256:a808f8bcd824acac4d82f9468a5b093eddc46a12468f99bab3c6191e9dd429ac
    Copying blob sha256:7f4f7d6e8dc6d3bc92ecb3f182a4177c46653fcf4a4e3eaecdebdebd6eebcf5d
    Copying blob sha256:3762fddd8cf345cbdd4413f62bf112b2a194c330803a1290e4582c26a357a2a6
    Copying blob sha256:f5624c74d9d1c71e06ad4ca643f473e837b6696d5790a1c8c6c34fdbd1c5a52d
    Copying blob sha256:047916eec6087fc7002822733b41d8b4f76e06cbfee36edee420f8e19730f858
    Copying blob sha256:b0c85be337e40513d2c7cdd5760f24f66aa9f4e5b092f8de6ee10fbdd8c2a3bf
executor >  local (4)
[ee/87a03a] process > checkRef (Checking GRCh38_latest_rna.fna.gz)        [100%] 1 of 1 ✔
[-        ] process > epinano_flow:splitReference                         -
[-        ] process > epinano_flow:splitBams                              -
[-        ] process > epinano_flow:indexReference                         -
[-        ] process > epinano_flow:EPINANO_CALC_VAR_FREQUENCIES           -
[-        ] process > epinano_flow:joinEpinanoRes                         -
[-        ] process > epinano_flow:makeEpinanoPlots_ins                   -
[-        ] process > epinano_flow:makeEpinanoPlots_mis                   -
[-        ] process > epinano_flow:makeEpinanoPlots_del                   -
[1f/508eec] process > compore_polish_flow:getChromInfo (reference.fa)     [100%] 1 of 1 ✔
[-        ] process > compore_polish_flow:NANOPOLISH_EVENTALIGN:index     -
[-        ] process > compore_polish_flow:NANOPOLISH_EVENTALIGN:eventa... -
[-        ] process > compore_polish_flow:NANOPOLISH_EVENTALIGN:eventa... -
[-        ] process > compore_polish_flow:mean_per_pos                    -
[-        ] process > compore_polish_flow:concat_mean_per_pos             -
[-        ] process > compore_polish_flow:concat_csv_files                -
[-        ] process > compore_polish_flow:NANOCOMPORE_SAMPLE_COMPARE:s... -
[-        ] process > tombo_common_flow:multiToSingleFast5                -
[-        ] process > tombo_common_flow:TOMBO_RESQUIGGLE_RNA:resquiggl... -
[d7/1632d4] process > getChromInfo (reference.fa)                         [100%] 1 of 1 ✔
[-        ] process > tombo_msc_flow:TOMBO_GET_MODIFICATION_MSC:getMod... -
[-        ] process > bedGraphToWig_msc                                   -
[-        ] process > tombo_lsc_flow:TOMBO_GET_MODIFICATION_LSC:getMod... -
[-        ] process > bedGraphToWig_lsc                                   -
[-        ] process > wigToBigWig                                         -
[-        ] process > mergeTomboWigsPlus                                  -
[-        ] process > mergeTomboWigsMinus                                 -
[-        ] process > EPINANO_VER:getVersion                              -
[-        ] process > NANOPOLISH_VER:getVersion                           -
[29/5e20d5] process > NANOCOMPORE_VER:getVersion                          [100%] 1 of 1 ✔
[-        ] process > TOMBO_VER:getVersion                                -
Error executing process > 'compore_polish_flow:NANOPOLISH_EVENTALIGN:index (mod)'

Caused by:
  Failed to pull singularity image
  command: singularity pull  --name biocorecrg-mopnanopolish-0.2.img.pulling.1669106082892 docker://biocorecrg/mopnanopolish:0.2 > /dev/null
  status : 255
  message:
    INFO:    Converting OCI blobs to SIF format
    INFO:    Starting build...
    Getting image source signatures
    Copying blob sha256:1e74cf3ea8b17b29912c91cc0ea80b5d3aed0ffe25aa2c8d3d4b24c1454f7052
    Copying blob sha256:e9092151f7a2491c0d2b42f59c1b8e2bf859ac6e53321eba7bb1017b2b3fe43f
    Copying blob sha256:276898f99f7b162c4834f6e13978f98bfd5f377635cdc1416d1fe27d38b47820
    Copying blob sha256:a521508fa11e33da609b8f0480253ef148b68b8beb56c0e409d0d207fda5a363
    Copying blob sha256:61af738b4846d9ef2e9eea025fb44288da5b8c76e8289a4f9eb62a2beb58626e
    Copying blob sha256:e518f22d2fe88cb8e3a813783655e7615b90d288ebd5e683e43b14c7769f69de
    Copying blob sha256:a808f8bcd824acac4d82f9468a5b093eddc46a12468f99bab3c6191e9dd429ac
    Copying blob sha256:7f4f7d6e8dc6d3bc92ecb3f182a4177c46653fcf4a4e3eaecdebdebd6eebcf5d
    Copying blob sha256:3762fddd8cf345cbdd4413f62bf112b2a194c330803a1290e4582c26a357a2a6
    Copying blob sha256:f5624c74d9d1c71e06ad4ca643f473e837b6696d5790a1c8c6c34fdbd1c5a52d
    Copying blob sha256:047916eec6087fc7002822733b41d8b4f76e06cbfee36edee420f8e19730f858
    Copying blob sha256:b0c85be337e40513d2c7cdd5760f24f66aa9f4e5b092f8de6ee10fbdd8c2a3bf
    Copying blob sha256:d5f76c9d11f315a83cab863e8042317dc05769b7bfa9603f7f09428fcd0b0978
    Copying blob sha256:7a9fddbdb3ca2e07ce37064c5ea6b42c0da570572c6c2580a153e2a6ced82b9d
    Copying blob sha256:07f993c72f4b1c62324d5d7ce799cef664e8b08c5df1e85c19f5dc4944db92c4
    Copying blob sha256:9385a7d7f89642d91499350880e6ab2a42bd31620bdce18cc307a10f8037475a
    Copying blob sha256:d766cc541a45a81ec4823b4522ee7f961b2d2ec98584774410ad992cade39fd9
    Copying blob sha256:ff4390cdd1a14ee619901d589ed9c24a2e79a58f42edb97ee27aa941357efa6d
    Copying blob sha256:dbd74f6531c6809f24ba5cf13f151dc94174a9848cbe1d2daacc3702d764a708
    Copying blob sha256:193e5a425e6baba6bb4d219a908e6e0e94fa4b69f2624446fb7240bc8adb38f6
    Copying blob sha256:a4d7254676fb442fa47db40c1b7e31bb0a953b3b52729a3b219916d06cbdd11e
    Copying blob sha256:682c399a9a404d74ee2fb669352db06c0583ba5b9a76f83e9bed0b590dbf7114
    Copying blob sha256:fa9f9b526b080e6f5c15fc671c80d3b7c3b84ddb57b4fd1db7f27d0368413d7d
    Copying blob sha256:7e98e8e2e69a1baea5a8e4df1fc657903304ed9cd8a8f84f98b24317bfe9c475
    Copying blob sha256:2f764843722a65951befd0232dc7c453142a11911667cce6e419ac3dff10fa26
    Copying blob sha256:23f4f9e84f759a26b8f6619ce5dfc8dc3bf1fd5409c7fb49e0aa69461a27483e
    Copying blob sha256:af7d7f43af91468f02bd674b094241b28dbd771b43e2ece3443910f4466f9a96
    Copying blob sha256:2ed7e2ef039b630733aa3cee0eaa454f5e98fa69d3c1d3f81fc0f57a4207030b
    Copying blob sha256:5cde6efba43bb788e3f92dca391ee09820293f0f3a906980a478e5db35468934
    Copying blob sha256:4a9b777eaf0f68990621a1bd66ff627211828315d1390632e70bc7511e247b1e
    Copying blob sha256:ecf6594c79b29b52d168274ad3abdb26d80e653abb7872cbebb9199807bd4c80
    Copying blob sha256:4ab6764dfb64dc73c156b165684a9ec8f06dc3034b08fc3a0b752fa911e55b67
    Copying blob sha256:75ae47fa2f8d70cc832e3dbb3bb63c62fde987cfd31b26cc657dda76948a9ac4
    Copying blob sha256:2c865baeaed15854e3093e5951773c1e9738c1bd40096c6132a4ff94e28950a9
    Copying blob sha256:9f302ca66cbf87680453ef9cb0f6c220730cf2473455dfce4e609947e095af82
    Copying blob sha256:70a97fbc76dd0537bcba4ae7b643a17d70fb7cc61daeaf5e98a13a754b2eb75f
    Copying blob sha256:ddf4e48a4ee60b47435494e1deeb8a333ed9ea4c8910b9fd4eed2c168617f46f
    Copying blob sha256:516e19bd299e96b54c7a1adf62f396ec543a19dfc09fb73383f52a277081eaf6
    Copying blob sha256:43cb3fe5c4aeae6df98c09fb0e43c8a92956f56349011d0d3d7063546a95221b
    Copying blob sha256:571b01b3ddffa820ebee3b1d97115356e039339699aabea91448c4c591070732
    Copying blob sha256:6f2eeb8410b2970ee873f72a1b0a935020363ef73bc490532725d5c539e5dfc9
    Copying config sha256:ec012d2e1e8779518bc0840db99d65a9b8ba084662ccbb9f457c09dbb6de27b2
    Writing manifest to image destination
    Storing signatures
    FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: conveyor failed to get: no descriptor found for reference "440b2c8927bbe2d4e64f19fdf5c304bad7fd7f9c70802cb25b07ec77d9243cf0"

ubuntu@ubuntu:~/MOP2/mop_mod$

params.config.test

params {
    input_path         = "/home/ubuntu/MOP2/mop_preprocess/output"
    comparison         = "$baseDir/comparison.tsv"

    reference           = "/home/ubuntu/MOP2/data/GRCh38_latest_rna.fna.gz"
    
    output             = "$baseDir/output"

    pars_tools         = "$baseDir/tools_opt.tsv"
    
    // flows 
    epinano       = "YES"
    nanocompore   = "YES"
    tombo_lsc     = "YES"
    tombo_msc     = "YES"

    // epinano plots
    epinano_plots = "YES"

    email              = ""
}

local.config

process {
	executor = 'local'
	cpus = 23
	memory = '30GB'    
    cache='lenient'
    container = 'biocorecrg/mopprepr:0.7'
    containerOptions = { workflow.containerEngine == "docker" ? '-u $(id -u):$(id -g) --platform amd64': null}
    withLabel: big_cpus_ignore {
        errorStrategy = 'ignore'
	
    }
    withLabel: basecall_gpus {
	    maxForks = 1
	    containerOptions = { workflow.containerEngine == "singularity" ? '--nv':
		   ( workflow.containerEngine == "docker" ? '-u $(id -u):$(id -g) --gpus all': null ) } 
    }
}

how to set core and threads number manually

Hi, I'm trying to run MOP2. But I met a problem. How can I set core and threads number manually? Thank you！

Error with Graphmap/samtools

Hello,

I get an error when I try to use mop_preprocess on my data (which are on fast5, so it's not supposed to be an issue). My reference is the human transcript ( https://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_43/gencode.v43.transcripts.fa.gz ).
I also ran it with docker and tried to switch to minimap2.
Thanks in advance for your answer.
log.zip

I have no output for mop_preprocess in the files : alignement, counts, cram_files (the files are empty).

I'm pretty sure this is due to the fact that the alignment with minimap2 is not working and that this may be caused by my data.
final_summary_FAW56164_db7c5011_600df0f2no_depletion.txt
maturemini.fa.gz
params.config.zip

docker files usage

Hi, I wanted to know how to use the docker files added in the repository?

container with guppy installed

From what I have seen, the guppy package is manually installed by downloading and placing under mop_preprocess/bin. Is there a reason why there is no singularity container that already has it?
I have run into issues using GPUs because the libraries could not be found.
If it's due to software license and distribution, maybe a script that creates that singularity image by using a supplied guppy url link should make setup easier

CPU test run with singularity/apptainer issue

@lucacozzuto Hello I am having trouble doing the intial test run. I only changed the GPU variable relative to the params.config.test file. The issue seems to be with singularity which is now called apptainer.

N E X T F L O W  ~  version 23.04.2
Launching `mop_preprocess.nf` [silly_bardeen] DSL2 - revision: ec40fe0af4


╔╦╗╔═╗╔═╗  ╔═╗┬─┐┌─┐┌─┐┬─┐┌─┐┌─┐┌─┐┌─┐┌─┐
║║║║ ║╠═╝  ╠═╝├┬┘├┤ ├─┘├┬┘│ ││  ├┤ └─┐└─┐
╩ ╩╚═╝╩    ╩  ┴└─└─┘┴  ┴└─└─┘└─┘└─┘└─┘└─┘
                                                                                       
====================================================
BIOCORE@CRG Master of Pores 2. Preprocessing - N F  ~  version 2.0
====================================================

conffile.                 : final_summary_01.txt

fast5                     : /tools/MOP2/mop_preprocess/../data/**/*.fast5
fastq                     : 

reference                 : /tools/MOP2/mop_preprocess/../anno/yeast_rRNA_ref.fa.gz
annotation                : 

granularity.              : 1

ref_type                  : transcriptome
pars_tools                : drna_tool_splice_opt.tsv

output                    : /tools/MOP2/mop_preprocess/output

GPU                       : OFF

basecalling               : guppy 
demultiplexing            : NO 
demulti_fast5             : NO

filtering                 : nanoq
mapping                   : graphmap

counting                  : nanocount
discovery                 : NO

cram_conv           	  : YES
subsampling_cram          : 50


saveSpace                 : NO
email                     : 

Skipping the email

----------------------CHECK TOOLS -----------------------------
basecalling : guppy
> demultiplexing will be skipped
mapping : graphmap
filtering : nanoq
counting : nanocount
> discovery will be skipped
--------------------------------------------------------------
[-        ] process > flow1:GUPPY_BASECALL:baseCall  -
[-        ] process > flow1:NANOQ_FILTER:filter      -
[-        ] process > preprocess_flow:MinIONQC       -
[-        ] process > preprocess_flow:RNA2DNA        -
[-        ] process > preprocess_flow:GRAPHMAP:map   -
[-        ] process > preprocess_flow:SAMTOOLS_CA... -
[-        ] process > preprocess_flow:SAMTOOLS_SO... -
[-        ] process > preprocess_flow:SAMTOOLS_IN... -
[-        ] process > preprocess_flow:checkRef       -
[-        ] process > preprocess_flow:bam2Cram       -
[-        ] process > preprocess_flow:bam2stats      -
[-        ] process > preprocess_flow:joinAlnStats   -
[-        ] process > preprocess_flow:NANOPLOT_QC... -
[-        ] process > preprocess_flow:concatenate... -
[-        ] process > preprocess_flow:FASTQC:fastQC  -
[-        ] process > preprocess_flow:NANOCOUNT:n... -
[-        ] process > preprocess_flow:AssignReads    -
[-        ] process > preprocess_flow:countStats     -
[-        ] process > preprocess_flow:joinCountStats -
[-        ] process > preprocess_flow:MULTIQC:mak... -

executor >  local (3)
[55/9a2d89] process > flow1:GUPPY_BASECALL:baseCa... [  0%] 0 of 2
[-        ] process > flow1:NANOQ_FILTER:filter      -
[-        ] process > preprocess_flow:MinIONQC       -
[-        ] process > preprocess_flow:RNA2DNA        -
[-        ] process > preprocess_flow:GRAPHMAP:map   -
[-        ] process > preprocess_flow:SAMTOOLS_CA... -
[-        ] process > preprocess_flow:SAMTOOLS_SO... -
[-        ] process > preprocess_flow:SAMTOOLS_IN... -
[eb/edc76d] process > preprocess_flow:checkRef (C... [  0%] 0 of 1
[-        ] process > preprocess_flow:bam2Cram       -
[-        ] process > preprocess_flow:bam2stats      -
[-        ] process > preprocess_flow:joinAlnStats   -
[-        ] process > preprocess_flow:NANOPLOT_QC... -
[-        ] process > preprocess_flow:concatenate... -
[-        ] process > preprocess_flow:FASTQC:fastQC  -
[-        ] process > preprocess_flow:NANOCOUNT:n... -
[-        ] process > preprocess_flow:AssignReads    -
[-        ] process > preprocess_flow:countStats     -
[-        ] process > preprocess_flow:joinCountStats -
[-        ] process > preprocess_flow:MULTIQC:mak... -

executor >  local (3)
[55/9a2d89] process > flow1:GUPPY_BASECALL:baseCa... [  0%] 0 of 2
[-        ] process > flow1:NANOQ_FILTER:filter      -
[-        ] process > preprocess_flow:MinIONQC       -
[-        ] process > preprocess_flow:RNA2DNA        -
[-        ] process > preprocess_flow:GRAPHMAP:map   -
[-        ] process > preprocess_flow:SAMTOOLS_CA... -
[-        ] process > preprocess_flow:SAMTOOLS_SO... -
[-        ] process > preprocess_flow:SAMTOOLS_IN... -
[eb/edc76d] process > preprocess_flow:checkRef (C... [  0%] 0 of 1
[-        ] process > preprocess_flow:bam2Cram       -
[-        ] process > preprocess_flow:bam2stats      -
[-        ] process > preprocess_flow:joinAlnStats   -
[-        ] process > preprocess_flow:NANOPLOT_QC... -
[-        ] process > preprocess_flow:concatenate... -
[-        ] process > preprocess_flow:FASTQC:fastQC  -
[-        ] process > preprocess_flow:NANOCOUNT:n... -
[-        ] process > preprocess_flow:AssignReads    -
[-        ] process > preprocess_flow:countStats     -
[-        ] process > preprocess_flow:joinCountStats -
[-        ] process > preprocess_flow:MULTIQC:mak... -
ERROR ~ Error executing process > 'flow1:GUPPY_BASECALL:baseCall (wt---1)'

Caused by:
  Process `flow1:GUPPY_BASECALL:baseCall (wt---1)` terminated with an error exit status (255)

Command executed:

  guppy_basecaller          --fast5_out --flowcell FLO-MIN106 --kit SQK-RNA002  -i ./         --save_path ./wt---1_out         --gpu_runners_per_device 1         --cpu_threads_per_caller 1 	    --num_callers  8
  cat wt---1_out/*.fastq >> wt---1.fastq
  rm wt---1_out/*.fastq
  gzip wt---1.fastq

Command exit status:
  255

Command output:
  (empty)

Command error:
  WARNING: DEPRECATED USAGE: Environment variable SINGULARITYENV_TMPDIR will not be supported in the future, use APPTAINERENV_TMPDIR instead
  INFO:    squashfuse not found, will not be able to mount SIF
  FATAL:   container creation failed: mount hook function failure: mount /proc/self/fd/3->/usr/local/biotools/apptainer/apptainer-1.1.4/var/apptainer/mnt/session/rootfs error: while mounting image /proc/self/fd/3: squashfuse not found

Work dir:
  /tools/MOP2/mop_preprocess/work/93/ce55d9f9007b705a980dd8733a3655

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

 -- Check '.nextflow.log' file for details
Pipeline BIOCORE@CRG Master of Pore - preprocess completed!
Started at  2023-07-18T13:29:56.808621-05:00
Finished at 2023-07-18T13:30:03.837118-05:00
Time elapsed: 7s
Execution status: failed

executor >  local (3)
[55/9a2d89] process > flow1:GUPPY_BASECALL:baseCa... [100%] 2 of 2, failed: 2 ✘
[-        ] process > flow1:NANOQ_FILTER:filter      -
[-        ] process > preprocess_flow:MinIONQC       -
[-        ] process > preprocess_flow:RNA2DNA        -
[-        ] process > preprocess_flow:GRAPHMAP:map   -
[-        ] process > preprocess_flow:SAMTOOLS_CA... -
[-        ] process > preprocess_flow:SAMTOOLS_SO... -
[-        ] process > preprocess_flow:SAMTOOLS_IN... -
[eb/edc76d] process > preprocess_flow:checkRef (C... [100%] 1 of 1, failed: 1 ✘
[-        ] process > preprocess_flow:bam2Cram       -
[-        ] process > preprocess_flow:bam2stats      -
[-        ] process > preprocess_flow:joinAlnStats   -
[-        ] process > preprocess_flow:NANOPLOT_QC... -
[-        ] process > preprocess_flow:concatenate... -
[-        ] process > preprocess_flow:FASTQC:fastQC  -
[-        ] process > preprocess_flow:NANOCOUNT:n... -
[-        ] process > preprocess_flow:AssignReads    -
[-        ] process > preprocess_flow:countStats     -
[-        ] process > preprocess_flow:joinCountStats -
[-        ] process > preprocess_flow:MULTIQC:mak... -
ERROR ~ Error executing process > 'flow1:GUPPY_BASECALL:baseCall (wt---1)'

Caused by:
  Process `flow1:GUPPY_BASECALL:baseCall (wt---1)` terminated with an error exit status (255)

Command executed:

  guppy_basecaller          --fast5_out --flowcell FLO-MIN106 --kit SQK-RNA002  -i ./         --save_path ./wt---1_out         --gpu_runners_per_device 1         --cpu_threads_per_caller 1 	    --num_callers  8
  cat wt---1_out/*.fastq >> wt---1.fastq
  rm wt---1_out/*.fastq
  gzip wt---1.fastq

Command exit status:
  255

Command output:
  (empty)

Command error:
  WARNING: DEPRECATED USAGE: Environment variable SINGULARITYENV_TMPDIR will not be supported in the future, use APPTAINERENV_TMPDIR instead
  INFO:    squashfuse not found, will not be able to mount SIF
  FATAL:   container creation failed: mount hook function failure: mount /proc/self/fd/3->/usr/local/biotools/apptainer/apptainer-1.1.4/var/apptainer/mnt/session/rootfs error: while mounting image /proc/self/fd/3: squashfuse not found

Work dir:
  /tools/MOP2/mop_preprocess/work/93/ce55d9f9007b705a980dd8733a3655

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

 -- Check '.nextflow.log' file for details

Problem with running mop_preprocess.nf

Hi,

There is a problem running mop_preprocess.nf.

(drna_mop2) prom@PCT0223:/path/MOP2/mop_preprocess$ /path/nextflow run mop_preprocess.nf -with-singularity -profile local
N E X T F L O W  ~  version 22.10.2
Launching `mop_preprocess.nf` [hopeful_majorana] DSL2 - revision: ec40fe0af4


╔╦╗╔═╗╔═╗  ╔═╗┬─┐┌─┐┌─┐┬─┐┌─┐┌─┐┌─┐┌─┐┌─┐
║║║║ ║╠═╝  ╠═╝├┬┘├┤ ├─┘├┬┘│ ││  ├┤ └─┐└─┐
╩ ╩╚═╝╩    ╩  ┴└─└─┘┴  ┴└─└─┘└─┘└─┘└─┘└─┘
                                                                                       
====================================================
BIOCORE@CRG Master of Pores 2. Preprocessing - N F  ~  version 2.0
====================================================

conffile.                 : final_summary_01.txt

fast5                     : 
fastq                     : /path/**/*.fastq

reference                 : /data/PublicData/refSeq_transcripts/GRCh38_latest_rna.fna
annotation                : /data/PublicData/refSeq_transcripts/GRCh38_latest_genomic.gff.gz

granularity.              : 1

ref_type                  : transcriptome
pars_tools                : drna_tool_splice_opt.tsv

output                    : /path/MOP2/mop_preprocess/output

GPU                       : ON

basecalling               : guppy 
demultiplexing            : NO 
demulti_fast5             : NO

filtering                 : nanoq
mapping                   : graphmap2

counting                  : nanocount
discovery                 : bambu

cram_conv           	  : NO
subsampling_cram          : 50


saveSpace                 : NO
email                     : [email protected]

Sending the email to [email protected]

----------------------CHECK TOOLS -----------------------------
> basecalling will be skipped
> demultiplexing will be skipped
mapping : graphmap2
filtering : nanoq
counting : nanocount
discovery : bambu
--------------------------------------------------------------
[-        ] process > preprocess_simple:FASTQC:fastQC            -
[-        ] process > preprocess_simple:GRAPHMAP2:map            -
[-        ] process > preprocess_simple:FASTQC:fastQC            -
[-        ] process > preprocess_simple:GRAPHMAP2:map            -
[-        ] process > preprocess_simple:SAMTOOLS_SORT:sortAln    -
[-        ] process > preprocess_simple:SAMTOOLS_INDEX:indexBam  -
[-        ] process > preprocess_simple:bam2stats                -
[-        ] process > preprocess_simple:joinAlnStats             -
[-        ] process > preprocess_simple:NANOPLOT_QC:MOP_nanoPlot -
[-        ] process > preprocess_simple:NANOCOUNT:nanoCount      -
[-        ] process > preprocess_simple:AssignReads              -
[-        ] process > preprocess_simple:countStats               -
[-        ] process > preprocess_simple:joinCountStats           -[-        ] process > preprocess_simple:MULTIQC:makeReport       -

[-        ] process > preprocess_simple:FASTQC:fastQC            -
[-        ] process > preprocess_simple:GRAPHMAP2:map            -
[-        ] process > preprocess_simple:SAMTOOLS_SORT:sortAln    -
[-        ] process > preprocess_simple:SAMTOOLS_INDEX:indexBam  -
[-        ] process > preprocess_simple:bam2stats                -
[-        ] process > preprocess_simple:joinAlnStats             -[-        ] process > preprocess_simple:NANOPLOT_QC:MOP_nanoPlot -
[-        ] process > preprocess_simple:NANOCOUNT:nanoCount      -
[-        ] process > preprocess_simple:AssignReads              -
[-        ] process > preprocess_simple:countStats               -
[-        ] process > preprocess_simple:joinCountStats           -
[-        ] process > preprocess_simple:MULTIQC:makeReport       -
Error executing process > 'preprocess_simple:FASTQC:fastQC (PAI52977_pass_d5246539_0.fastq)'

Caused by:
  Failed to pull singularity image
  command: singularity pull  --name quay.io-biocontainers-fastqc-0.11.9--0.img.pulling.1668531299614 docker://quay.io/biocontainers/fastqc:0.11.9--0 > /dev/null
  status : 255
  message:
    INFO:    Converting OCI blobs to SIF format
    INFO:    Starting build...
    Getting image source signatures
    Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
    Copying blob sha256:77c6c00e8b61bb628567c060b85690b0b0561bb37d8ad3f3792877bddcfe2500
    Copying blob sha256:3aaade50789a6510c60e536f5e75fe8b8fc84801620e575cb0435e2654ffd7f6
    Copying blob sha256:00cf8b9f3d2a08745635830064530c931d16f549d031013a9b7c6535e7107b88
    Copying blob sha256:7ff999a2256f84141f17d07d26539acea8a4d9c149fefbbcc9a8b4d15ea32de7
    Copying blob sha256:d2ba336f2e4458a9223203bf17cc88d77e3006d9cbf4f0b24a1618d0a5b82053
    Copying blob sha256:dfda3e01f2b637b7b89adb401f2f763d592fcedd2937240e2eb3286fabce55f0
    Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
    Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
    Copying blob sha256:10c3bb32200bdb5006b484c59b5f0c71b4dbab611d33fca816cd44f9f5ce9e3c
    Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
    Copying blob sha256:6d92b3a49ebfad5fe895550c2cb24b6370d61783aa4f979702a94892cbd19077
    Copying config sha256:8b79910fc535b5dd09ef33662f939e7e51006d1ef5545c919e72cc877acbe5b1
    Writing manifest to image destination
    Storing signatures
    2022/11/15 16:55:38  info unpack layer: sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
    2022/11/15 16:55:39  info unpack layer: sha256:77c6c00e8b61bb628567c060b85690b0b0561bb37d8ad3f3792877bddcfe2500    2022/11/15 16:55:39  warn rootless{dev/console} creating empty file in place of device 5:1    2022/11/15 16:55:39  info unpack layer: sha256:3aaade50789a6510c60e536f5e75fe8b8fc84801620e575cb0435e2654ffd7f6
    2022/11/15 16:55:39  info unpack layer: sha256:00cf8b9f3d2a08745635830064530c931d16f549d031013a9b7c6535e7107b88
    2022/11/15 16:55:39  info unpack layer: sha256:7ff999a2256f84141f17d07d26539acea8a4d9c149fefbbcc9a8b4d15ea32de7
    2022/11/15 16:55:39  info unpack layer: sha256:d2ba336f2e4458a9223203bf17cc88d77e3006d9cbf4f0b24a1618d0a5b82053
    2022/11/15 16:55:39  info unpack layer: sha256:dfda3e01f2b637b7b89adb401f2f763d592fcedd2937240e2eb3286fabce55f0
    2022/11/15 16:55:39  info unpack layer: sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
    2022/11/15 16:55:39  info unpack layer: sha256:10c3bb32200bdb5006b484c59b5f0c71b4dbab611d33fca816cd44f9f5ce9e3c
    2022/11/15 16:55:39  info unpack layer: sha256:6d92b3a49ebfad5fe895550c2cb24b6370d61783aa4f979702a94892cbd19077
    INFO:    Creating SIF file...
    FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: while creating SIF: while creating container: writing data object for SIF file: copying data object file to SIF file: write /home/prom/.singularity/cache/oci-tmp/tmp_223233459: no space left on device

(drna_mop2) prom@PCT0223:/path/MOP2/mop_preprocess$

(drna_mop2) prom@PCT0223:/path/MOP2/mop_preprocess$ which singularity
/usr/local/bin/singularity
(drna_mop2) prom@PCT0223:/path/MOP2/mop_preprocess$ singularity version
3.6.3
(drna_mop2) prom@PCT0223:/path/MOP2/mop_preprocess$ guppy_basecaller -v
: Guppy Basecalling Software, (C) Oxford Nanopore Technologies, Limited. Version 3.4.5+fb1fbfb
(drna_mop2) prom@PCT0223:/path/MOP2/mop_preprocess$ which guppy_basecaller
/path/MOP2/mop_preprocess/bin/ont-guppy/bin/guppy_basecaller

I attach the configuration file params.config.test:

params {
    conffile            = "final_summary_01.txt"
    fast5               = ""
    fastq               = "/path/**/*.fastq"

    reference           = "/data/PublicData/refSeq_transcripts/GRCh38_latest_rna.fna"
    annotation          = "/data/PublicData/refSeq_transcripts/GRCh38_latest_genomic.gff.gz"
    ref_type            = "transcriptome"

    pars_tools          = "drna_tool_splice_opt.tsv" 
    output              = "$baseDir/output"
    qualityqc           = 5
    granularity         = 1

    basecalling         = "guppy"
    GPU                 = "ON"
    demultiplexing      = "NO"
    demulti_fast5       = "NO" 

    filtering           = "nanoq"

    mapping             = "graphmap2"
    counting            = "nanocount"
    discovery           = "bambu"

    cram_conv           = "NO"
    subsampling_cram    = 50

    saveSpace           = "NO"

    email               = "[email protected]"
}

singularity pull / credentials for quay.io

Error executing process > 'flow1:NANOQ_FILTER:filter (wt---1)'

Caused by:
  Failed to pull singularity image
  command: singularity pull  --name quay.io-biocontainers-nanoq-0.8.2--h779adbc_0.img.pulling.1647379514424 docker://quay.io/biocontainers/nanoq:0.8.2--h779adbc_0 > /dev/null
  status : 255
  message:
    FATAL:   While making image from oci registry: error fetching image to cache: failed to get checksum for docker://quay.io/biocontainers/nanoq:0.8.2--h779adbc_0: unable to retrieve auth token: invalid username/password: unauthorized: Invalid Username or Password

Upon seeing this error I got user/pass from quay.io. I can now manually build these images. But when I run nextflow:
nextflow run mop_preprocess.nf -with-singularity -bg -profile slurm
I see the same authentication error again. How do I pass my quay.io credentials to nextflow for creating singularity images?

no output for Nanoconsensus

Hi Luca,
first of all thanks for this great resource! It makes my life much easier. I have run the pipeline with the test data and when I reach the mop_consensus module, it does not produce any output folder, although it say that the process is completed. Please see below:

N E X T F L O W ~ version 22.10.6
Launching mop_consensus.nf [elegant_lichterman] DSL2 - revision: d4ddf1d0a4

╔╦╗╔═╗╔═╗ ╔═╗╔═╗╔╗╔╔═╗╔═╗╔╗╔╔═╗╦ ╦╔═╗
║║║║ ║╠═╝ ║ ║ ║║║║╚═╗║╣ ║║║╚═╗║ ║╚═╗
╩ ╩╚═╝╩ ╚═╝╚═╝╝╚╝╚═╝╚═╝╝╚╝╚═╝╚═╝╚═╝

====================================================
BIOCORE@CRG Master of Pores 2. Get consensus modifications - N F ~ version 2.0

***************** Input *********************
input_path : /home/jason_s/bioinf_isilon/Research/TOMAZOU/Internal/jason/Nanopore/MOP2/mop_consensus/../mop_mod/output_mod
output : /home/jason_s/bioinf_isilon/Research/TOMAZOU/Internal/jason/Nanopore/MOP2/mop_consensus/output
comparison : /home/jason_s/bioinf_isilon/Research/TOMAZOU/Internal/jason/Nanopore/MOP2/mop_consensus/comparison.tsv
padsize : 50

******* reference has to be the genome **********
reference : /home/jason_s/bioinf_isilon/Research/TOMAZOU/Internal/jason/Nanopore/MOP2/mop_consensus/../anno/yeast_rRNA_ref.fa.gz
email : yourname@yourdomain

Sending the email to yourname@yourdomain

[- ] process > checkRef -
[- ] process > indexFasta -

executor > local (1)
[7a/77d56d] process > checkRef (Checking yeast_rRNA_ref.fa.gz) [ 0%] 0 of 1
[- ] process > indexFasta -
[- ] process > nanoConsensus -

executor > local (1)
[7a/77d56d] process > checkRef (Checking yeast_rRNA_ref.fa.gz) [100%] 1 of 1 ✔
[- ] process > indexFasta [ 0%] 0 of 1
[- ] process > nanoConsensus -

executor > local (2)
[7a/77d56d] process > checkRef (Checking yeast_rRNA_ref.fa.gz) [100%] 1 of 1 ✔
[53/c6fc4c] process > indexFasta (reference.fa) [ 0%] 0 of 1
[- ] process > nanoConsensus -

executor > local (2)
[7a/77d56d] process > checkRef (Checking yeast_rRNA_ref.fa.gz) [100%] 1 of 1 ✔
[53/c6fc4c] process > indexFasta (reference.fa) [100%] 1 of 1 ✔
[- ] process > nanoConsensus -
Pipeline BIOCORE@CRG Master of Pore completed!
Started at 2023-04-04T15:17:29.342723+02:00
Finished at 2023-04-04T15:17:35.874643+02:00
Time elapsed: 6.5s
Execution status: OK
Failed to invoke workflow.onComplete event handler

-- Check script 'mop_consensus.nf' at line: 158 or see '.nextflow.log' file for more details

Guppy parameter issue : --fast5_out

Hi,
I'm trying to troubleshoot an error i keep receiving the command ran is : nextflow run mop_preprocess.nf -with-singularity > log_profound.txt

Log file :----------------------CHECK TOOLS -----------------------------
basecalling : guppy
demultiplexing : deeplexicon

mapping will be skipped
filtering will be skipped
counting will be skipped
discovery will be skipped

[- ] process > flow2:GUPPY_BASECALL:baseCall -
[- ] process > flow2:DEMULTIPLEX_DEEPLEXICON:demultiplex -

[- ] process > flow2:GUPPY_BASECALL:baseCall -
[- ] process > flow2:DEMULTIPLEX_DEEPLEXICON:demultiplex -
[- ] process > flow2:extracting_demultiplexed_fastq -
[- ] process > preprocess_flow:MinIONQC -
[- ] process > preprocess_flow:concatenateFastQFiles -
[- ] process > preprocess_flow:FASTQC:fastQC -
[- ] process > preprocess_flow:MULTIQC:makeReport -
Pulling Singularity image docker://lpryszcz/deeplexicon:1.2.0 [cache /home/apps/MasterOFPores/MOP2/mop_preprocess/../singularity/lpryszcz-deeplexicon-1.2.0.img]

executor > local (8)
[b5/dd2bc5] process > flow2:GUPPY_BASECALL:baseCall (fast5---6) [ 0%] 0 of 14
[- ] process > flow2:DEMULTIPLEX_DEEPLEXICON:demultiplex -
[- ] process > flow2:extracting_demultiplexed_fastq -
[- ] process > preprocess_flow:MinIONQC -
[- ] process > preprocess_flow:concatenateFastQFiles -
[- ] process > preprocess_flow:FASTQC:fastQC -
[- ] process > preprocess_flow:MULTIQC:makeReport -
Pulling Singularity image docker://lpryszcz/deeplexicon:1.2.0 [cache /home/apps/MasterOFPores/MOP2/mop_preprocess/../singularity/lpryszcz-deeplexicon-1.2.0.img]

executor > local (9)
[3d/106346] process > flow2:GUPPY_BASECALL:baseCall (fast5---9) [ 0%] 0 of 14
[- ] process > flow2:DEMULTIPLEX_DEEPLEXICON:demultiplex -
[- ] process > flow2:extracting_demultiplexed_fastq -
[- ] process > preprocess_flow:MinIONQC -
[- ] process > preprocess_flow:concatenateFastQFiles -
[- ] process > preprocess_flow:FASTQC:fastQC -
[- ] process > preprocess_flow:MULTIQC:makeReport -
Pulling Singularity image docker://lpryszcz/deeplexicon:1.2.0 [cache /home/apps/MasterOFPores/MOP2/mop_preprocess/../singularity/lpryszcz-deeplexicon-1.2.0.img]
Error executing process > 'flow2:GUPPY_BASECALL:baseCall (fast5---5)'

Caused by:
Process flow2:GUPPY_BASECALL:baseCall (fast5---5) terminated with an error exit status (255)

Command executed:

guppy_basecaller --fast5_out --flowcell FLO-MIN106 --kit SQK-RNA002 --disable_qscore_filtering -i ./ --save_path ./fast5---5_out --gpu_runners_per_device 1 --cpu_threads_per_caller 1 --num_callers 8
cat fast5---5_out/.fastq >> fast5---5.fastq
rm fast5---5_out/.fastq
gzip fast5---5.fastq

Command exit status:
255

Command output:
(empty)

Command error:
INFO: Converting SIF file to temporary sandbox...
WARNING: underlay of /etc/localtime required more than 50 (82) bind mounts
Unexpected token '--fast5_out' on command-line.
Try 'guppy_basecaller --help' for more information.
INFO: Cleaning up image...

Work dir:
/home/apps/MasterOFPores/MOP2/mop_preprocess/work/ee/5859310d8a589e9d3616cbf10c19f8

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named .command.sh

Pipeline BIOCORE@CRG Master of Pore - preprocess completed!
Started at 2023-02-08T16:13:28.084-05:00
Finished at 2023-02-08T16:13:52.036-05:00
Time elapsed: 24s
Execution status: failed
Failed to invoke workflow.onComplete event handler

-- Check script 'mop_preprocess.nf' at line: 632 or see '.nextflow.log' file for more details
WARN: Killing pending tasks (8)

executor > local (9)
[3d/106346] process > flow2:GUPPY_BASECALL:baseCall (fast5---9) [ 16%] 1 of 6, failed: 1
[- ] process > flow2:DEMULTIPLEX_DEEPLEXICON:demultiplex -
[- ] process > flow2:extracting_demultiplexed_fastq -
[- ] process > preprocess_flow:MinIONQC -
[- ] process > preprocess_flow:concatenateFastQFiles -
[- ] process > preprocess_flow:FASTQC:fastQC -
[- ] process > preprocess_flow:MULTIQC:makeReport -
Pulling Singularity image docker://lpryszcz/deeplexicon:1.2.0 [cache /home/apps/MasterOFPores/MOP2/mop_preprocess/../singularity/lpryszcz-deeplexicon-1.2.0.img]
Error executing process > 'flow2:GUPPY_BASECALL:baseCall (fast5---5)'

Caused by:
Process flow2:GUPPY_BASECALL:baseCall (fast5---5) terminated with an error exit status (255)

Command executed:

Command exit status:
255

Command output:
(empty)

Work dir:
/home/apps/MasterOFPores/MOP2/mop_preprocess/work/ee/5859310d8a589e9d3616cbf10c19f8

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named .command.sh

epinano version 1.1 or 1.2

There are two versions of epinano implemented in MOP2: 1.1 and 1.2, and I think 1.1 is being used by default. The package authors of epinano mention 1.2 seems to perform better.
Is the epinano 1.2 in MOP2 fully implemented or still in development? Is there a particular reason why it's not the default?

RE: Reference sequences

Hi,

In the mop_tail and mop_mod workflow, what the Reference option refers to? The genome or the transcriptome or whatever reference was used with the mop_preprocessing step?

is it possible to run mop_mod on one sample, i.e. no comparisons. If yes, what to do about the comparison.tsv file?

Many thanks

Searching for RNA modifications (m6A, Y, m5C, etc).

Good day, colleagues,

I have a question regarding the search for several RNA modifications, including:

N6-methyladenosine (m6A)
pseudouridine (Y)
5-methylcytosine (m5C)
5-hydroxymethylcytosine (hm5C)
1-methyladenosine (m1A)
N3-methylcytosine (m3C)
N4-acetylcytosine (ac4C)
7-methylguanosine (m7G)

Can I search for these modifications using the Master of Pores 2? Does nanocompore allow you to search when running mop_mod.nf? If yes, what reference value should be used?

What references can be used at all? The references from Piano DB (http://180.208.58.19/%ce%a8-WHISTLE/index.html) and RMVar (https://rmvar.renlab.org/download.html) do not fit the format.

So far, I am only using the full human transcriptome - GRCh38_latest_rna.fna and the annotation GRCh38_latest_genomic.gff.gz.

Could you help to search for the above modifications?

biocorecrg / mop2 Goto Github PK

mop2's Introduction

MoP2- DSL2 version of Master of Pores

Install

Testing

Reference

Documentation

mop2's People

Stargazers

Watchers

Forkers

mop2's Issues

==================================================== BIOCORE@CRG Master of Pores 2. Preprocessing - N F ~ version 2.0

Command exit status:

==================================================== BIOCORE@CRG Master of Pores 2. Preprocessing - N F ~ version 2.0

==================================================== BIOCORE@CRG Master of Pores 2. Preprocessing - N F ~ version 2.0

==================================================== BIOCORE@CRG Master of Pores 2. Preprocessing - N F ~ version 2.0

==================================================== BIOCORE@CRG Master of Pores 2. Get consensus modifications - N F ~ version 2.0

Recommend Projects

Recommend Topics

Recommend Org

====================================================
BIOCORE@CRG Master of Pores 2. Preprocessing - N F ~ version 2.0

====================================================
BIOCORE@CRG Master of Pores 2. Preprocessing - N F ~ version 2.0

====================================================
BIOCORE@CRG Master of Pores 2. Preprocessing - N F ~ version 2.0

====================================================
BIOCORE@CRG Master of Pores 2. Preprocessing - N F ~ version 2.0

====================================================
BIOCORE@CRG Master of Pores 2. Get consensus modifications - N F ~ version 2.0