error with minpath_running directory about picrust2 HOT 15 CLOSED

Hi Gavin,
the commands are as follows:

With qiime2 plugin (works)

qiime picrust2 full-pipeline \
    --i-table features/table.qza \
    --i-seq features/rep-seqs.qza \
    --p-threads $SLURM_NTASKS \
    --output-dir PICRUSt2 \
   --verbose

With the full pipeline (errno 29, errno 17)

picrust2_pipeline.py \
  --study_fasta dna-sequences.fasta \
  --input feature-table.biom \
  --output pipeline \
  --stratified \
  --coverage \
  --per_sequence_contrib \
  --threads $SLURM_NTASKS \
  --verbose

With the Pathway step alone (errno 17):

pathway_pipeline.py \
  -i EC_metagenome_out/pred_metagenome_unstrat.tsv \
  -o pathways_out \
  --intermediate pathways_working \
  -p $SLURM_NTASKS \
  --coverage \
  --per_sequence_contrib \
  --per_sequence_abun EC_metagenome_out/seqtab_norm.tsv \
  --per_sequence_function EC_predicted.tsv

$SLURM_NTASKS = 24

These are the error codes:
Errno 29 gets thrown during the hsp.py stage

Traceback (most recent call last):
File "[...]/anaconda3/envs/picrust2/bin/picrust2_pipeline.py", line 205, in
main()
File "[...]/anaconda3/envs/picrust2/bin/picrust2_pipeline.py", line 198, in main
verbose=args.verbose)
File "[...]/software/anaconda3/envs/picrust2/lib/python3.6/site-packages/picrust2/pipeline.py", line 173, in full_pipeline
system_call_check(hsp_cmd, print_out=verbose)
File "[...]/software/anaconda3/envs/picrust2/lib/python3.6/site-packages/picrust2/util.py", line 247, in system_call_check
print(f.read(), file=sys.stderr)
File "[...]/software/anaconda3/envs/picrust2/lib/python3.6/tempfile.py", line 807, in exit
self.cleanup()
File "[...]/software/anaconda3/envs/picrust2/lib/python3.6/tempfile.py", line 811, in cleanup
_shutil.rmtree(self.name)
File "[...]/software/anaconda3/envs/picrust2/lib/python3.6/shutil.py", line 484, in rmtree
onerror(os.rmdir, path, sys.exc_info())
File "[...]/software/anaconda3/envs/picrust2/lib/python3.6/shutil.py", line 482, in rmtree
os.rmdir(path)
OSError: [Errno 39] Directory not empty: [...]/tmp/tmpkc_gfm80'

And I temporarily patched by adding a pass at line 482 in the shutil.py

    try:
        if os.path.samestat(orig_st, os.fstat(fd)):
            _rmtree_safe_fd(fd, path, onerror)
            try:
                os.rmdir(path)
            except OSError:
                pass
                #onerror(os.rmdir, path, sys.exc_info())

Errno 17 gets thrown at pathway_pipeline.py

Traceback (most recent call last):
File "[...]/anaconda3/envs/picrust2/bin/pathway_pipeline.py", line 197, in
main()
File "[...]/anaconda3/envs/picrust2/bin/pathway_pipeline.py", line 155, in main
print_cmds=args.print_cmds)
File "[...]/anaconda3/envs/picrust2/lib/python3.6/site-packages/picrust2/pathway_pipeline.py", line 466, in pathway_pipeline
print_opt=print_cmds)
File "[...]/anaconda3/envs/picrust2/lib/python3.6/site-packages/picrust2/pathway_pipeline.py", line 687, in per_sequence_contrib_levels
for sequence in overlapping_sequence_ids)
File "[...]/anaconda3/envs/picrust2/lib/python3.6/site-packages/joblib/parallel.py", line 934, in call
self.retrieve()
File "[...]/anaconda3/envs/picrust2/lib/python3.6/site-packages/joblib/parallel.py", line 833, in retrieve
self._output.extend(job.get(timeout=self.timeout))
File "[...]/anaconda3/envs/picrust2/lib/python3.6/site-packages/joblib/_parallel_backends.py", line 521, in wrap_future_result
return future.result(timeout=timeout)
File "[...]/anaconda3/envs/picrust2/lib/python3.6/concurrent/futures/_base.py", line 432, in result
return self.__get_result()
File "[...]/anaconda3/envs/picrust2/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
OSError: Could not create directory '%s'. Are permissions set correctly? Got error: '[Errno 17] File exists: '[...]/PICRUSt2/stratified/pathways_working/per_sequence_contrib/minpath_running''

That I temporarily patched by commenting out line 384 in _base.py as:

def __get_result(self):
    #if self._exception:
       #raise self._exception
    #else:
       #return self._result
    return self._result

With those patches, I obtained the results I needed, but I know it's a temporary solution.
Thank you!

from picrust2.

Hi there,

Would you mind posting the full command you're trying to run and what version of PICRUSt2 you are using? It looks like the directory for intermediate files already exists for some reason - hopefully it will work if you specify a different output directory.

from picrust2.

Sorry, thought I had posted my command!
I am running it on my university's cluster, so have to load a few modules prior to being able to actually run the command I want:

module purge
module load miniconda gcc-latest
source activate qiime2-2018.8
cd /data/c.older/FIV_ging
picrust2_pipeline.py -s picrust2/filtered_rep-sequences.fasta -i picrust2/filtered_feature-table.biom -o picrust2_pipe_output_12042018_2 --threads 10 -n
source deactivate

I had tried running this (including moving my input files) from a new directory, and that did not resolve the issue. And just to be clear, I did not create any other directories within the directory where my input files were and where I was running it.

from picrust2.

Thanks for posting the command. Just to clarify - did this error occur after you updated to a different version of PICRUSt2 or just when you tried to re-run a command after it worked the first time? And it sounds like you'll get the same error if you specify a different output folder than picrust2_pipe_output_12042018_2, correct? What version of PICRUSt2 are you running? You can type: picrust2_pipeline.py --version

from picrust2.

I didn't update PICRUSt2 (using picrust2_pipeline.py 2.0.3-b).
The error occured when I tried to re-run, which wasn't immediately after the initial successful run, but ~month later with the same exact code (had saved it from the first run, so am confident I am not writing anything incorrectly). With there being a lapse of 1 month, it does seem like it could be something to do with the version of this or some other supporting software, but was not smart enough to record what these versions were and don't believe I have any way to see what these are :(
Correct, I get the same error even if I specify a different output folder.

from picrust2.

Ok thanks. My guess is that an incompatibility slipped into that QIIME2 environment at some point. I would recommend re-installing PICRUSt2 in a new conda environment following the instructions here: https://github.com/picrust/picrust2/wiki/Installation

Note that you shouldn't need to re-install EPA-NG and GAPPA!

from picrust2.

Looks like we are still getting the same error, even after re-installing :(
Will try to find a way to run this outside of the cluster, but any other suggestions?

from picrust2.

Sorry to hear that! You could try running each step individually as described here

from picrust2.

Sorry it's taken me so long to respond, got sidetracked by some other work! I ended up trying on another cluster using the qiime2 plugin because this seemed like an easier fix for me :)
Thank you very much for your help!

from picrust2.

Well that's good to hear!

from picrust2.

Having the same problem on an HPC cluster, fresh conda install of version 2.1.0-b.

I also encountered error 29, which I patched by commenting out line 482 of the shutil.py and add a "pass" at the OSError exception.

It seems there are issues with files/folder not being released? Changing permission to the temp folder or the intermediate folders via chmod -R 777 does not solve the problem.

Problem happens both when running the full picrust2_pipeline.py or the single pathway_pipeline.py command. No problems with the qiime2 plug-in (but I was looking forward having the stratified outputs)

from picrust2.

Hi @andreanuzzo,

Could you share the exact commands you were running? Did you try running the command on just a single CPU?

from picrust2.

Thanks for the details on these issues. I haven't been able to reproduce the issues myself, but I don't have access to a cluster. Nonetheless I have made a couple of changes that I just pushed to the master. Would you mind downloading the latest PICRUSt2 development version and seeing if these errors are resolved?

Best,

Gavin

from picrust2.

The full pipeline script worked perfectly now. Thank you!
(can you please push to conda too?)

from picrust2.

Good to hear! I'll push to conda once I have a new release ready.

from picrust2.