DADA2 split regions singularity about ampliseq HOT 13 CLOSED

Hi there, that sounds bad! Could you test the following commands whether they make any problems (error should occur early at startup):

nextflow run nf-core/ampliseq -r 2.9.0 -profile test -config /home/bq_boezay/bin/nextflow_config.yaml  --outdir "home/bq_boezay/documents/pilot_test"
nextflow run nf-core/ampliseq -r 2.9.0 -profile test_full -config /home/bq_boezay/bin/nextflow_config.yaml  --outdir "home/bq_boezay/documents/pilot_test"

I removed -profile singularity because you have that in the config already. I added -profile test / test_full because they represent reproducible tests.
If those fail, you have a corrupted copy of the pipeline. Rinse and repeat.
If those pass, and when you try your previously failed command again and it fails again, then we have to dig deeper.

from ampliseq.

@d4straub Thank you for your answer.

When I try the commands as they are, I get the following message:
Project nf-core/ampliseq contains uncommitted changes -- Cannot switch to revision: 2.9.0

And when I try them without the -r 2.9.0, I get the same error as before:
ERROR ~ No such file or directory: Can't find a matching module file for include: ../modules/local/dada2_splitregions

So I should erase everything and install again? Do you have any tips for installing, because it actually seems pretty straightforward with:
curl -s https://get.nextflow.io | bash
chmod +x nextflow
mv nextflow ~/bin/

And that's what I had done, but still the pipeline copy seems to be corrupted. What is to stop it from happening again?

from ampliseq.

I cleaned everything and re-installed nextflow:

curl -s https://get.nextflow.io | bash
chmod +x nextflow
mv nextflow ~/bin/
export PATH="$HOME/bin:$PATH"
nextflow info

Version: 23.10.1 build 5891
Created: 12-01-2024 22:01 UTC (22:01 GMT)
System: Linux 3.10.0-1160.108.1.el7.x86_64
Runtime: Groovy 3.0.19 on OpenJDK 64-Bit Server VM 11.0.22+7-LTS
Encoding: UTF-8 (ANSI_X3.4-1968)

And when I run the test code you have given me @d4straub, the error message is the same.

nextflow run nf-core/ampliseq -profile test -config /home/bq_boezay/bin/nextflow_config.yaml --outdir "home/bq_boezay/documents/pilot_test"

ERROR ~ No such file or directory: Can't find a matching module file for include: ../modules/local/dada2_splitregions

from ampliseq.

Ok, we identified the problem.

Project nf-core/ampliseq contains uncommitted changes -- Cannot switch to revision: 2.9.0

means that you have a modified corrupted copy. Re-installing nextflow doesnt help, because it doesnt remove the global cache, by default located in ~/.nextflow/assets. The solution is essentially to remove cache, see nf-core/rnaseq#505. rm -rf ~/.nextflow/assets/nf-core/ampliseq should do the trick (untested, cannot verify right now).

from ampliseq.

Okay thank you @d4straub, I did that, now we have a different kind of error:

ERROR ~ Error executing process > 'NFCORE_AMPLISEQ:AMPLISEQ:METADATA_PAIRWISE (Metadata_full.tsv)

Caused by:
Process NFCORE_AMPLISEQ:AMPLISEQ:METADATA_PAIRWISE (Metadata_full.tsv) terminated with an error exit status (127)

Command executed:

metadata_pairwise.r Metadata_full.tsv

cat <<-END_VERSIONS > versions.yml
"NFCORE_AMPLISEQ:AMPLISEQ:METADATA_PAIRWISE":
R: $(R --version 2>&1 | sed -n 1p | sed 's/R version //' | sed 's/ (.*//')
END_VERSIONS

Command exit status:
127

Command output:
(empty)

Command error:
/usr/bin/env: Rscript: No such file or directory

Work dir:
/home/bq_boezay/work/41/e33077d2dd3887bd8fcb199100ab07

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named .command.sh

-- Check '.nextflow.log' file for details
ERROR ~ Unexpected error [AbortedException]

-- Check script '.nextflow/assets/nf-core/ampliseq/./workflows/ampliseq.nf' at line: 322 or see '.nextflow.log' file for more details

from ampliseq.

Hm, I assume that was with one of the commands that I recommended? That error indicates that Rscript couldnt be found, which points towards a missing software, i.e. container problem. I assume furthermore that no process was successfully executed and that this error occurred with one of the first processes that started. If all that assumptions are true, try with -profile test,singularity (i.e. add singularity), if that helps, then your config isnt working as expected. If that doesnt help, then your singularity installation might be broken.

from ampliseq.

Hi @binnuor, I found that you used (and I blindly copied it in my recommended test commands) -config /home/bq_boezay/bin/nextflow_config.yaml while I had previously thought you used the file https://github.com/nf-core/ampliseq/files/15012700/nextflow_config.txt that you shared. But nextflow_config.yaml isnt the same as nextflow_config.txt. Therefore you might use other settings than I assumed. If the problem isnt resolved, please mention again the complete command and share all configs so that troubleshooting is based on complete information.

from ampliseq.

Hello @d4straub, sadly different errors keep occurring with FastQC or QIIME2 when I try to run. I had uploaded the config file I use above, but here it is again:

process {
    executor = 'slurm'
    queue    = 'single'
    cpus = 8
    time = '12h'
    memory = '200GB'
    singularity = true
    conda = false
    slurm = true
    default_resources = [
          "slurm_account=ag-saez",
          "slurm_partition=single"
        ]
}

By adding singularity to the command, I did get farther but still haven't achieved an errorless run with the following:

nextflow run nf-core/ampliseq \
	-resume \
	-r 2.9.0 \
	-profile test,singularity \
	-config /home/bq_boezay/bin/nextflow_config.yaml \
	--outdir "home/bq_boezay/documents/pilot_test"

from ampliseq.

Thats an issue with singularity setup and/or your config.
The code of your config you pasted here seems unusual to me, partially even wrong. Check out configs in https://nf-co.re/configs. Lets try to modify the config. Please use the following code in a file named my_ampliseg.config (not yaml):

process {
    executor = 'slurm'
    queue    = 'single'
    max_cpus = 8
    max_time = 12.h
    max_memory = 200.GB
}

(I hope default_resources isnt needed, I dont know and have never seen that part.)
and use with

nextflow run nf-core/ampliseq \
	-resume \
	-r 2.9.0 \
	-profile test,singularity \
	-c my_ampliseg.config \
	--outdir "home/bq_boezay/documents/pilot_test"

from ampliseq.

@d4straub I have tried again with the my_ampliseg.config file you suggested me, like this:

nextflow run nf-core/ampliseq \
> -resume \
> -r 2.9.0 \
> -profile test,singularity \
> -c bin/my_ampliseg.config \
> --outdir "home/bq_boezay/documents/pilot_test"

There are multiple errors following:

ERROR ~ Error executing process > 'NFCORE_AMPLISEQ:AMPLISEQ:QIIME2_EXPORT:QIIME2_EXPORT_ABSOLUTE'

Caused by:
Process NFCORE_AMPLISEQ:AMPLISEQ:QIIME2_EXPORT:QIIME2_EXPORT_ABSOLUTE terminated with an error exit status (132)

Command executed:

export XDG_CONFIG_HOME="./xdgconfig"
export MPLCONFIGDIR="./mplconfigdir"
export NUMBA_CACHE_DIR="./numbacache"

#produce raw count table in biom format "table/feature-table.biom"
qiime tools export
--input-path filtered-table.qza
--output-path table
cp table/feature-table.biom .

#produce raw count table "table/feature-table.tsv"
biom convert
-i table/feature-table.biom
-o feature-table.tsv
--to-tsv

#produce representative sequence fasta file "sequences.fasta"
qiime feature-table tabulate-seqs
--i-data filtered-sequences.qza
--o-visualization rep-seqs.qzv
qiime tools export
--input-path rep-seqs.qzv
--output-path representative_sequences
cp representative_sequences/sequences.fasta rep-seq.fasta
cp representative_sequences/*.tsv .

##on several taxa level
array=($(seq 2 1 4))
for i in ${array[@]}
do
#collapse taxa
qiime taxa collapse
--i-table filtered-table.qza
--i-taxonomy taxonomy.qza
--p-level $i
--o-collapsed-table table-$i.qza
#export to biom
qiime tools export
--input-path table-$i.qza
--output-path table-$i
#convert to tab separated text file
biom convert
-i table-$i/feature-table.biom
-o abs-abund-table-$i.tsv --to-tsv
done

cat <<-END_VERSIONS > versions.yml
"NFCORE_AMPLISEQ:AMPLISEQ:QIIME2_EXPORT:QIIME2_EXPORT_ABSOLUTE":
qiime2: $( qiime --version | sed '1!d;s/.* //' )
END_VERSIONS

Command exit status:
132

Command output:
(empty)

ERROR ~ Error executing process > 'NFCORE_AMPLISEQ:AMPLISEQ:QIIME2_EXPORT:QIIME2_EXPORT_ABSOLUTE'

Caused by:
Process NFCORE_AMPLISEQ:AMPLISEQ:QIIME2_EXPORT:QIIME2_EXPORT_ABSOLUTE terminated with an error exit status (132)

Command executed:

export XDG_CONFIG_HOME="./xdgconfig"
export MPLCONFIGDIR="./mplconfigdir"
export NUMBA_CACHE_DIR="./numbacache"

#produce raw count table in biom format "table/feature-table.biom"
qiime tools export
--input-path filtered-table.qza
--output-path table
cp table/feature-table.biom .

#produce raw count table "table/feature-table.tsv"
biom convert
-i table/feature-table.biom
-o feature-table.tsv
--to-tsv

#produce representative sequence fasta file "sequences.fasta"
qiime feature-table tabulate-seqs
--i-data filtered-sequences.qza
--o-visualization rep-seqs.qzv
qiime tools export
--input-path rep-seqs.qzv
--output-path representative_sequences
cp representative_sequences/sequences.fasta rep-seq.fasta
cp representative_sequences/*.tsv .

##on several taxa level
array=($(seq 2 1 4))
for i in ${array[@]}
do
#collapse taxa
qiime taxa collapse
--i-table filtered-table.qza
--i-taxonomy taxonomy.qza
--p-level $i
--o-collapsed-table table-$i.qza
#export to biom
qiime tools export
--input-path table-$i.qza
--output-path table-$i
#convert to tab separated text file
biom convert
-i table-$i/feature-table.biom
-o abs-abund-table-$i.tsv --to-tsv
done

cat <<-END_VERSIONS > versions.yml
"NFCORE_AMPLISEQ:AMPLISEQ:QIIME2_EXPORT:QIIME2_EXPORT_ABSOLUTE":
qiime2: $( qiime --version | sed '1!d;s/.* //' )
END_VERSIONS

Command exit status:
132

Command output:
(empty)

Command error:
INFO: Converting SIF file to temporary sandbox...
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)
.command.run: line 153: awk: command not found
.command.run: line 153: head: command not found
.command.run: line 154: awk: command not found
.command.run: line 156: tr: command not found
.command.run: line 156: head: command not found
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)
.command.run: line 86: awk: command not found
ps: error while loading shared libraries: libsystemd.so.0: cannot open shared object file: No such file or directory
.command.run: line 30: awk: command not found
.command.run: line 31: awk: command not found
.command.run: line 32: tr: command not found
.command.run: line 33: awk: command not found
.command.run: line 34: awk: command not found
.command.run: line 35: awk: command not found
ps: error while loading shared libraries: libsystemd.so.0: cannot open shared object file: No such file or directory
.command.run: line 30: awk: command not found
.command.run: line 31: awk: command not found
.command.run: line 32: tr: command not found
.command.run: line 33: awk: command not found
.command.run: line 34: awk: command not found
.command.run: line 35: awk: command not found
ps: error while loading shared libraries: libsystemd.so.0: cannot open shared object file: No such file or directory
.command.run: line 30: awk: command not found
.command.run: line 31: awk: command not found
.command.run: line 32: tr: command not found
.command.run: line 33: awk: command not found
.command.run: line 34: awk: command not found
.command.run: line 35: awk: command not found
ps: error while loading shared libraries: libsystemd.so.0: cannot open shared object file: No such file or directory
.command.run: line 30: awk: command not found
.command.run: line 31: awk: command not found
.command.run: line 32: tr: command not found
.command.run: line 33: awk: command not found
.command.run: line 34: awk: command not found
.command.run: line 35: awk: command not found
ps: error while loading shared libraries: libsystemd.so.0: cannot open shared object file: No such file or directory
.command.run: line 30: awk: command not found
.command.run: line 31: awk: command not found
.command.run: line 32: tr: command not found
.command.run: line 33: awk: command not found
.command.run: line 34: awk: command not found
.command.run: line 35: awk: command not found
.command.sh: line 9: 41 Illegal instruction qiime tools export --input-path filtered-table.qza --output-path table
INFO: Cleaning up image...

Work dir:
/home/bq_boezay/work/b6/b3b40f521dab9c873002300045a3a6

Tip: view the complete command output by changing to the process work dir and entering the command cat .command.out

-- Check '.nextflow.log' file for details

from ampliseq.

Problems are your server settings and/or config in play with your server. I am out of options to give advice. Turn to your sys admin.

from ampliseq.

Hello @d4straub, we figured out the problem. It seems that ampliseq wasn't compatible with slurm in the cluster somehow, within an interactive session and also on a different cluster without slurm, it ran without any problems. Just wanted to let you know, incase someone else deals with this issue. Thank you so much for your help and efforts. Have a nice weekend :)

from ampliseq.

Alright, thanks. I use ampliseq on several clusters with slurm successfully. Anyway, glad it works for you now!

from ampliseq.