Comments (15)
Hi Gavin,
the commands are as follows:
With qiime2 plugin (works)
qiime picrust2 full-pipeline \
--i-table features/table.qza \
--i-seq features/rep-seqs.qza \
--p-threads $SLURM_NTASKS \
--output-dir PICRUSt2 \
--verbose
With the full pipeline (errno 29, errno 17)
picrust2_pipeline.py \
--study_fasta dna-sequences.fasta \
--input feature-table.biom \
--output pipeline \
--stratified \
--coverage \
--per_sequence_contrib \
--threads $SLURM_NTASKS \
--verbose
With the Pathway step alone (errno 17):
pathway_pipeline.py \
-i EC_metagenome_out/pred_metagenome_unstrat.tsv \
-o pathways_out \
--intermediate pathways_working \
-p $SLURM_NTASKS \
--coverage \
--per_sequence_contrib \
--per_sequence_abun EC_metagenome_out/seqtab_norm.tsv \
--per_sequence_function EC_predicted.tsv
$SLURM_NTASKS = 24
These are the error codes:
Errno 29 gets thrown during the hsp.py stage
Traceback (most recent call last):
File "[...]/anaconda3/envs/picrust2/bin/picrust2_pipeline.py", line 205, in
main()
File "[...]/anaconda3/envs/picrust2/bin/picrust2_pipeline.py", line 198, in main
verbose=args.verbose)
File "[...]/software/anaconda3/envs/picrust2/lib/python3.6/site-packages/picrust2/pipeline.py", line 173, in full_pipeline
system_call_check(hsp_cmd, print_out=verbose)
File "[...]/software/anaconda3/envs/picrust2/lib/python3.6/site-packages/picrust2/util.py", line 247, in system_call_check
print(f.read(), file=sys.stderr)
File "[...]/software/anaconda3/envs/picrust2/lib/python3.6/tempfile.py", line 807, in exit
self.cleanup()
File "[...]/software/anaconda3/envs/picrust2/lib/python3.6/tempfile.py", line 811, in cleanup
_shutil.rmtree(self.name)
File "[...]/software/anaconda3/envs/picrust2/lib/python3.6/shutil.py", line 484, in rmtree
onerror(os.rmdir, path, sys.exc_info())
File "[...]/software/anaconda3/envs/picrust2/lib/python3.6/shutil.py", line 482, in rmtree
os.rmdir(path)
OSError: [Errno 39] Directory not empty: [...]/tmp/tmpkc_gfm80'
And I temporarily patched by adding a pass
at line 482 in the shutil.py
try: if os.path.samestat(orig_st, os.fstat(fd)): _rmtree_safe_fd(fd, path, onerror) try: os.rmdir(path) except OSError: pass #onerror(os.rmdir, path, sys.exc_info())
Errno 17 gets thrown at pathway_pipeline.py
Traceback (most recent call last):
File "[...]/anaconda3/envs/picrust2/bin/pathway_pipeline.py", line 197, in
main()
File "[...]/anaconda3/envs/picrust2/bin/pathway_pipeline.py", line 155, in main
print_cmds=args.print_cmds)
File "[...]/anaconda3/envs/picrust2/lib/python3.6/site-packages/picrust2/pathway_pipeline.py", line 466, in pathway_pipeline
print_opt=print_cmds)
File "[...]/anaconda3/envs/picrust2/lib/python3.6/site-packages/picrust2/pathway_pipeline.py", line 687, in per_sequence_contrib_levels
for sequence in overlapping_sequence_ids)
File "[...]/anaconda3/envs/picrust2/lib/python3.6/site-packages/joblib/parallel.py", line 934, in call
self.retrieve()
File "[...]/anaconda3/envs/picrust2/lib/python3.6/site-packages/joblib/parallel.py", line 833, in retrieve
self._output.extend(job.get(timeout=self.timeout))
File "[...]/anaconda3/envs/picrust2/lib/python3.6/site-packages/joblib/_parallel_backends.py", line 521, in wrap_future_result
return future.result(timeout=timeout)
File "[...]/anaconda3/envs/picrust2/lib/python3.6/concurrent/futures/_base.py", line 432, in result
return self.__get_result()
File "[...]/anaconda3/envs/picrust2/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
OSError: Could not create directory '%s'. Are permissions set correctly? Got error: '[Errno 17] File exists: '[...]/PICRUSt2/stratified/pathways_working/per_sequence_contrib/minpath_running''
That I temporarily patched by commenting out line 384 in _base.py
as:
def __get_result(self): #if self._exception: #raise self._exception #else: #return self._result return self._result
With those patches, I obtained the results I needed, but I know it's a temporary solution.
Thank you!
from picrust2.
Hi there,
Would you mind posting the full command you're trying to run and what version of PICRUSt2 you are using? It looks like the directory for intermediate files already exists for some reason - hopefully it will work if you specify a different output directory.
from picrust2.
Sorry, thought I had posted my command!
I am running it on my university's cluster, so have to load a few modules prior to being able to actually run the command I want:
module purge
module load miniconda gcc-latest
source activate qiime2-2018.8
cd /data/c.older/FIV_ging
picrust2_pipeline.py -s picrust2/filtered_rep-sequences.fasta -i picrust2/filtered_feature-table.biom -o picrust2_pipe_output_12042018_2 --threads 10 -n
source deactivate
I had tried running this (including moving my input files) from a new directory, and that did not resolve the issue. And just to be clear, I did not create any other directories within the directory where my input files were and where I was running it.
from picrust2.
Thanks for posting the command. Just to clarify - did this error occur after you updated to a different version of PICRUSt2 or just when you tried to re-run a command after it worked the first time? And it sounds like you'll get the same error if you specify a different output folder than picrust2_pipe_output_12042018_2, correct? What version of PICRUSt2 are you running? You can type: picrust2_pipeline.py --version
from picrust2.
I didn't update PICRUSt2 (using picrust2_pipeline.py 2.0.3-b).
The error occured when I tried to re-run, which wasn't immediately after the initial successful run, but ~month later with the same exact code (had saved it from the first run, so am confident I am not writing anything incorrectly). With there being a lapse of 1 month, it does seem like it could be something to do with the version of this or some other supporting software, but was not smart enough to record what these versions were and don't believe I have any way to see what these are :(
Correct, I get the same error even if I specify a different output folder.
from picrust2.
Ok thanks. My guess is that an incompatibility slipped into that QIIME2 environment at some point. I would recommend re-installing PICRUSt2 in a new conda environment following the instructions here: https://github.com/picrust/picrust2/wiki/Installation
Note that you shouldn't need to re-install EPA-NG and GAPPA!
from picrust2.
Looks like we are still getting the same error, even after re-installing :(
Will try to find a way to run this outside of the cluster, but any other suggestions?
from picrust2.
Sorry to hear that! You could try running each step individually as described here
from picrust2.
Sorry it's taken me so long to respond, got sidetracked by some other work! I ended up trying on another cluster using the qiime2 plugin because this seemed like an easier fix for me :)
Thank you very much for your help!
from picrust2.
Well that's good to hear!
from picrust2.
Having the same problem on an HPC cluster, fresh conda install of version 2.1.0-b.
I also encountered error 29, which I patched by commenting out line 482 of the shutil.py and add a "pass" at the OSError exception.
It seems there are issues with files/folder not being released? Changing permission to the temp folder or the intermediate folders via chmod -R 777 does not solve the problem.
Problem happens both when running the full picrust2_pipeline.py or the single pathway_pipeline.py command. No problems with the qiime2 plug-in (but I was looking forward having the stratified outputs)
from picrust2.
Hi @andreanuzzo,
Could you share the exact commands you were running? Did you try running the command on just a single CPU?
from picrust2.
Thanks for the details on these issues. I haven't been able to reproduce the issues myself, but I don't have access to a cluster. Nonetheless I have made a couple of changes that I just pushed to the master. Would you mind downloading the latest PICRUSt2 development version and seeing if these errors are resolved?
Best,
Gavin
from picrust2.
The full pipeline script worked perfectly now. Thank you!
(can you please push to conda too?)
from picrust2.
Good to hear! I'll push to conda once I have a new release ready.
from picrust2.
Related Issues (20)
- UserWarning: A worker stopped while some jobs were given to the executor. HOT 14
- Error in predicting functions using fungal ITS data HOT 4
- Generating a KEGG module table from the KO table HOT 1
- place_seqs.py --ref_dir questions HOT 2
- FutureWarning / deprecation warnings HOT 1
- Error in runing picrust 2 HOT 2
- COG3512 was eliminated in the latest COG database (2020) HOT 1
- Retrieving more Fungi genomes for ITS and 18S predictions HOT 3
- Error picrust2 picrust2_pipeline.py HOT 3
- i get "1 failed, 60 passed" when i run the tests to verify the installation after installed picrust2-2.5.0 from source HOT 12
- Invalid model name error HOT 3
- PICRUSt2 with other database(e.g. cowpi) HOT 2
- warning line 341 "metagenome_pipeline.py" HOT 1
- Unable to install v2.5.1 on MackBook M1 with Monterey HOT 11
- Running PICRUSt on ITS sequences HOT 1
- Calling in ITS reference HOT 5
- Getting Reverse Complementary Seqs HOT 1
- q2-picrust2 Tutorial error_ "pro_ref.raxml_info" could not be found. HOT 1
- "Stopping - no ASV ids overlap between input FASTA and sequence abundance table" despite having exact ASV id matching. HOT 3
- PICRUSt2 Data and STAMP HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from picrust2.