Comments (5)
hi Daniel,
looks like there is a bug in parsing of FragGeneScan output. I haven't
encountered this before, my best guess is FragGeneScan crashed and
metAMOS tried to parse an empty file and failed. either way we should
handle this better. that said, could you please copy+paste the FINDORFS
log file in the ./Logs directory? Thanks!
best,
Todd
Hi folks,
I was happy to get the pipeline to run end to end on sub-sampling of my data to 10M paired reads.
I then attempted to run it on the entire data set of 76M paired reads.
It unfortunately crashed at the findORFS step.The error log is at bottom of this message.
Here are my additional questions:
- Is there a flag for printing the list of commands to a file or to STDOUT / STDERR ?
- what does the --fastest flag do specifically?
Here is the command used:
${metAMOS}/runPipeline -c amphora2 -d METAMOS_BS27FULL -g fraggenescan -k 43 -p 22 -a velvet 1> METAMOS_BS27FULL.run.out 2> METAMOS_BS27FULL.run.err&Her is the STDERR log:
Job = [[SGI_BS27.1.fastq, SGI_BS27.2.fastq] -> preprocess.success] completed
Completed Task = preprocess.Preprocess
Job = [[lib1.seq] -> [proba.asm.contig]] completed
Completed Task = assemble.Assemble
Job = [proba.asm.contig -> proba.bout] completed
Completed Task = mapreads.MapReads
Traceback (most recent call last):
File "/bioinformatics/asm/bio_bin/metAMOS/metAMOS-6b17a08-0.35/runPipeline", line 367, in
pipeline_run([preprocess.Preprocess,assemble.Assemble,findorfs.FindORFS, findreps.FindRepeats, annotate.Annotate, abundance.Abundance, scaffold.Scaffold, findscforfs.FindScaffoldORFS, propagate.Propagate, classify.Classify, postprocess.Postprocess], verbose = 1)
File "/bioinformatics/asm/bio_bin/metAMOS/metAMOS-6b17a08-0.35/Utilities/ruffus/task.py", line 2680, in pipeline_run
raise errt
ruffus.ruffus_exceptions.RethrownJobError:Exceptions running jobs for 'def findorfs.FindORFS(...):' Original exception: Exception #1 exceptions.ValueError(need more than 1 value to unpack): for findorfs.FindORFS.Job = [proba.asm.contig -> proba.faa] Traceback (most recent call last): File "/bioinformatics/asm/bio_bin/metAMOS/metAMOS-6b17a08-0.35/Utilities/ruffus/task.py", line 524, in run_pooled_job_without_exceptions return t_job_result(task_name, JOB_COMPLETED, job_name, return_value, None) File "/bio_bin/python26/lib/python2.6/contextlib.py", line 34, in __exit__ self.gen.throw(type, value, traceback) File "/bioinformatics/asm/bio_bin/metAMOS/metAMOS-6b17a08-0.35/Utilities/ruffus/task.py", line 232, in do_nothing_semaphore yield File "/bioinformatics/asm/bio_bin/metAMOS/metAMOS-6b17a08-0.35/Utilities/ruffus/task.py", line 517, in run_pooled_job_without_exceptions return_value = job_wrapper(param, user_defined_work_func, register_cleanup, touch_files_only) File "/bioinformatics/asm/bio_bin/metAMOS/metAMOS-6b17a08-0.35/Utilities/ruffus/task.py", line 447, in job_wrapper_io_files ret_val = user_defined_work_func(*param) File "/bioinformatics/asm/bio_bin/metAMOS/metAMOS-6b17a08-0.35/src/findorfs.py", line 243, in FindORFS parse_fraggenescanout("%s/FindORFS/out/%s.orfs"%(_settings.rundir,_settings.PREFIX)) File "/bioinformatics/asm/bio_bin/metAMOS/metAMOS-6b17a08-0.35/src/findorfs.py", line 191, in parse_fraggenescanout hdr,gene = seq.split("\n",1) ValueError: need more than 1 value to unpack
Thanks
Reply to this email directly or view it on GitHub:
#45
from metamos.
Thanks for quick response - i have encountered this problem a few times before and Ido recall you warning me about the stability of FragGeneScan in this pipeline;
Here is the content of the FINDORFS.log:
unlink: cannot unlink `/home/dbrami/tmp/metAmos/METAMOS_BS27FULL/FindORFS/in/proba.asm.contig': No such file or directory
Here is the content of the findORFS folder:
FindORFS/
|-- in
| -- proba.asm.contig -> /home/dbrami/tmp/metAmos/METAMOS_BS27FULL/Assemble/out/proba.asm.contig
-- out
|-- proba.gene.cvg
|-- proba.orfs
|-- proba.orfs.faa
`-- proba.orfs.ffn
cmd-> ls -lstrh FindORFS/out/
total 821M
463M -rw-rw-r-- 1 dbrami employees 462M Apr 11 21:38 proba.orfs.ffn
221M -rw-rw-r-- 1 dbrami employees 221M Apr 11 21:38 proba.orfs.faa
139M -rw-rw-r-- 1 dbrami employees 139M Apr 11 21:38 proba.orfs
0 -rw-rw-r-- 1 dbrami employees 0 Apr 11 21:38 proba.gene.cvg
And for good measure the tree of the Assembly folder:
Assemble/
|-- in
-- out |-- Graph2 |-- IDX.1.ebwt |-- IDX.2.ebwt |-- IDX.3.ebwt |-- IDX.4.ebwt |-- IDX.rev.1.ebwt |-- IDX.rev.2.ebwt |-- LastGraph |-- Log |-- PreGraph |-- Roadmaps |-- Sequences |-- contigs.fa |-- contigs_wo_location_info.txt |-- proba.afg -> /home/dbrami/tmp/metAmos/METAMOS_BS27FULL/Assemble/out/velvet_asm.afg |-- proba.asm.contig -> /home/dbrami/tmp/metAmos/METAMOS_BS27FULL/Assemble/out/contigs.fa |-- proba.asm.tigr |-- proba.bout |-- proba.contig.cvg |-- proba.lib1.badmates |-- proba.lib1.hdr |-- proba.lib1.mappedmates |-- proba.lib1.mates_in_diff_contigs |-- proba.seq100.contig |-- stats.txt
-- velvet_asm.afg
from metamos.
Thanks for the addl info. so it looks like a FragGeneScan parsing error
since the files are not empty. would you mind sending me the files or
making them available via FTP so that I may debug this? The files I need
are the ones in :
FindORFS/out/
best,
Todd
On Thu, Apr 12, 2012 at 1:26 PM, dbrami <
[email protected]
wrote:
Thanks for quick response - i have encountered this problem a few times
before and Ido recall you warning me about the stability of FragGeneScan
in this pipeline;
Here is the content of the FINDORFS.log:unlink: cannot unlink
`/home/dbrami/tmp/metAmos/METAMOS_BS27FULL/FindORFS/in/proba.asm.contig':
No such file or directoryHere is the content of the findORFS folder:
FindORFS/
|-- in
|-- proba.asm.contig -> /home/dbrami/tmp/metAmos/METAMOS_BS27FULL/Assemble/out/proba.asm.contig
-- out
|-- proba.gene.cvg
|-- proba.orfs
|-- proba.orfs.faa
`-- proba.orfs.ffncmd-> ls -lstrh FindORFS/out/
total 821M
463M -rw-rw-r-- 1 dbrami employees 462M Apr 11 21:38 proba.orfs.ffn
221M -rw-rw-r-- 1 dbrami employees 221M Apr 11 21:38 proba.orfs.faa
139M -rw-rw-r-- 1 dbrami employees 139M Apr 11 21:38 proba.orfs
0 -rw-rw-r-- 1 dbrami employees 0 Apr 11 21:38 proba.gene.cvgAnd for good measure the tree of the Assembly folder:
Assemble/
|-- in
-- out |-- Graph2 |-- IDX.1.ebwt |-- IDX.2.ebwt |-- IDX.3.ebwt |-- IDX.4.ebwt |-- IDX.rev.1.ebwt |-- IDX.rev.2.ebwt |-- LastGraph |-- Log |-- PreGraph |-- Roadmaps |-- Sequences |-- contigs.fa |-- contigs_wo_location_info.txt |-- proba.afg -> /home/dbrami/tmp/metAmos/METAMOS_BS27FULL/Assemble/out/velvet_asm.afg |-- proba.asm.contig -> /home/dbrami/tmp/metAmos/METAMOS_BS27FULL/Assemble/out/contigs.fa |-- proba.asm.tigr |-- proba.bout |-- proba.contig.cvg |-- proba.lib1.badmates |-- proba.lib1.hdr |-- proba.lib1.mappedmates |-- proba.lib1.mates_in_diff_contigs |-- proba.seq100.contig |-- stats.txt
-- velvet_asm.afg
Reply to this email directly or view it on GitHub:
#45 (comment)
Todd J. Treangen, Ph.D.
Postdoctoral Fellow
McKusick-Nathans Institute of Genetic Medicine
Johns Hopkins University School of Medicine
Office: Bloomberg School of Public Health, E3138
615 N Wolfe St, Baltimore MD, 21205
Phone: 443-287-8782, FAX: 410-955-0958
Email: [email protected]
from metamos.
Thanks Todd,
Since I am working with sensitive data, the higher ups are uneasy about me sending anything. Let me attempt to re-run the pipeline again starting from the FindORF step.
Also, I dont have access to an FTP site where I can easily drop the data.
But I can say this, the crash seemed to have been very abrupt; notice the sequence ID has been truncated:
cmd-> tail proba.orfs
NODE_3889587_length_93_cov_1.021505
1 135 - 2 1.247610 I: D:
NODE_3889589_length_88_cov_1.340909
1 130 + 1 1.333019 I: D:
NODE_3889590_length_107_cov_1.102804
1 149 + 2 1.385543 I: D:
NODE_3889595_length_117_cov_1.008547
NODE_3889596_length_43_cov_1.651163
1 85 + 2 1.269308 I: D:
NODE_3889599_len
same with the other two files:
cmd-> tail proba.orfs.faa
PDRKQASQIDRYRLVIVDECSMINEELW
NODE_3889557_length_43_cov_1.000000_1_85_-
ASLMAPSLLDRVFLTRSKKRKADDEIQ
NODE_3889558_length_43_cov_1.000000_1_85_-
DGKKKTPKSVCPDGWSDFKNSLWARFST
NODE_3889559_length_43_cov_1.000000_1_85_+
TRQPLYNRQTIAHPGWTREAIRPSVRV
NODE_3889561_length_43_cov_1.000000_1_85_-
EMCGNGIRCMAKFSEALETQDGQPPQA
NODE_3889562_length_43_cov_
cmd-> tail proba.orfs.ffn
NODE_3889575_length_43_cov_1.000000_1_85_+
TGGGTTAAAAAACGATACATTGATCCTGCACCCCCCAACAAAGGCTTTCAAGGTGCTGATGGAATCTCGCGGAAATTCATC
NODE_3889576_length_43_cov_1.0
from metamos.
The latest code in the repository includes a verbose option (-v) to runPipeline that will print every command run to the stdout. Additionally, there is a file named Logs/COMMANDS.txt that lists every COMMAND run for each step of the pipeline.This issue should be addressed.
from metamos.
Related Issues (20)
- Some html reports are empty after running HOT 1
- Run test_phylosift.sh error HOT 9
- rm: cannot remove `~/all.seq.mates': No such file or directory HOT 2
- Some questions about the DB
- RunPipeline: project dir does not exist! HOT 1
- Deprecated dependencies HOT 1
- Failed at PREPROCESS with error code -6 HOT 2
- How to configure the settings for different tools
- Error when running run_pipeline_test.sh (FragGeneScan, exit code -11) HOT 1
- syntax error when execute python2 INSTALL.py core
- Steps to skip
- not recognizing changes to parameters HOT 2
- Troubles with a Docker implementation of iMetAMOS HOT 2
- **ERROR** All input sequences were empty HOT 4
- python INSTALL.py error HOT 1
- Install Phylosift Issue
- Assembly always runs out of memory
- [REAPR pipeline] Error opening file for writing ... task_pipeline.pl line 112. HOT 1
- metAMOS INSTALL.py error HOT 3
- metamos install error
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from metamos.