• Adding a fastqc process for input fastq files
• Adding a multiqc process to summarize fastqc reports for all samples

Metadata files should be provided with the --input parameter

Currently there is process logging as part of the pipeline already, but the logs need to be parsed for extracting the number and percentage of sequences that passed the process and the ones that were filetered out.

Add the python script to parse the logs as part of the pipeline.

Hey there are a couple of linting tests failing still in Travis, the rest passed!

INFO: ===========
LINTING RESULTS

82 tests passed 2 tests had warnings 2 tests failed
WARNING: Test Warnings:
http://nf-co.re/errors#4: Config variable not found: params.reads
http://nf-co.re/errors#4: Config variable not found: params.singleEnd
ERROR: Test Failures:
http://nf-co.re/errors#8: Conda environment name is incorrect (nf-core-bcellmagic-1.0, should be nf-core-bcellmagic-1.0dev)
http://nf-co.re/errors#8: Conda dependency did not have pinned version number: bioconda::biopython
ERROR: Sorry, some tests failed - exiting with a non-zero error code...

Just noticed this when trying to get the logo for bcellmagic

Hi.

I'm a new user and I'm trying to follow the Takara Bio TCR workflow.
However I'm having trouble with memory requirements.
I'm getting the following error:

Error executing process > 'NFCORE_BCELLMAGIC:BCELLMAGIC:FETCH_DATABASES (IMGT IGBLAST)'

Caused by:
Process requirement exceed available memory -- req: 12 GB; avail: 8 GB

Command executed:

fetch_databases.sh
echo $(date "+%F") > IMGT.version.txt

Command exit status:

Command output:
(empty)

Work dir:
/Volumes/Hard drive/bcellmagic/work/db/c293fe251fb5decf6087e4940e22a7

Tip: view the complete command output by changing to the process work dir and entering the command cat .command.out

Would you have any ideas please?

Thanks so much.
Aislinn.

It would be nice to include the clonal table to the output of EnchantR to be able to pass it to the users as well.

Description of the bug

There is an error in the Dowser lineages process when running the pipeline with the test profile (uses r-enchantr 0.0.3)

Command used and terminal output

run nf-core/airrflow -r airrflow -profile test,docker --outdir "results" -resume


label: unnamed-chunk-2
Quitting from lines 124-159 (_main.Rmd)
Error in buildIgphyml(data, igphyml = exec, temp_path = file.path(dir,  :
  The file /usr/local/share/igphyml/src/igphyml cannot be executed.
Calls: <Anonymous> ... eval_with_user_handlers -> eval -> eval -> getTrees -> buildIgphyml
In addition: Warning messages:
1: replacing previous import ‘data.table::last’ by ‘dplyr::last’ when loading ‘enchantr’
2: replacing previous import ‘data.table::first’ by ‘dplyr::first’ when loading ‘enchantr’
3: replacing previous import ‘data.table::between’ by ‘dplyr::between’ when loading ‘enchantr’
4: In buildIgphyml(data, igphyml = exec, temp_path = file.path(dir,  :
  Dowser igphyml doesn't mask split codons!

### Relevant files

_No response_

### System information

nextflow version 22.10.2.5832

Which made us run into trouble already.... right before Christmas!! :-(

Hi There!

I wanted to test out some cloud instances using nf-core airrflow. I launched the test pipeline on my laptop and that worked just fine.

But when I wanted to port this to the cloud, the test pipeline fails at the GUNZIP step. I noticed that the AWS results page is also not showing any results. Maybe this issue is similar on AWS?

Description of the bug

Airrflow test pipeline fails at the GUNZIP workflow on Google Cloud. It fails because it can not find the FASTQ files that it needs to unzip.

Steps to reproduce

Steps to reproduce the behaviour:

1. Command line:
   nextflow run nf-core/airrflow -r 2.0.0 -profile test,google --google_bucket gs://my-bucket
2. See error:

Error executing process > 'NFCORE_BCELLMAGIC:BCELLMAGIC:GUNZIP (Sample3)'

Caused by:
  Process `NFCORE_BCELLMAGIC:BCELLMAGIC:GUNZIP (Sample3)` terminated with an error exit status (9)

Command executed:

  gunzip -f "Sample3_UMI_R1.fastq.gz"
  gunzip -f "Sample3_R2.fastq.gz"
  echo $(gunzip --version 2>&1) | sed 's/^.*(gzip) //; s/ Copyright.*$//' > gunzip.version.txt

Command exit status:
  9

Command output:
  (empty)

Command error:
  Execution failed: generic::failed_precondition: while running "nf-986be5683bbd72136fc79e9bc86b254e-main": unexpected exit status 1 was not ignored

Expected behaviour

Successful completion of the gunzip step.

Log files

Here is the .command.log file of the relevant process:

+ cd /light-test/e3/d02332d9d12ca6f93ddf5b85b853ad
+ gsutil -m -q cp gs://my-bucket//light-test/e3/d02332d9d12ca6f93ddf5b85b853ad/command.run .
+ bash .command.run nxf_stage
+ [[ '' -gt 0 ]]
+ true
+ cd /light-test/e3/d02332d9d12ca6f93ddf5b85b853ad
+ bash .command.run nxf_unstage
CommandException: No URLs matched: .command.out
CommandException: 1 file/object could not be transferred.
CommandException: No URLs matched: .command.err
CommandException: 1 file/object could not be transferred.
CommandException: No URLs matched: .command.trace
CommandException: 1 file/object could not be transferred.
CommandException: No URLs matched: .exitcode
CommandException: 1 file/object could not be transferred.
ls: cannot access '*_R1.fastq': No such file or directory
ls: cannot access '*_R2.fastq': No such file or directory
ls: cannot access '*.version.txt': No such file or directory

Nextflow Installation

21.10.6

Description of the bug

As input samplesheet only supports single_cell FALSE, autogenerate this instead of requiring column in samplesheet.

Command used and terminal output

No response

Relevant files

No response

System information

No response

At startup, I got a vsearch missing error, and installed that tool. Turns out however that ClusterSets.py still uses usearch.

I would suggest that either the tools are converted to vsearch (possibly the best option?), or that a usearch version check is performed at the beginning.

currently, there are a lot of subfoders in the results folder. I want to organize this to have only the following subfolders:

• preprocessing
• clonal_analysis
• repertoire analysis
• multiQC
• pipeline_info

So main.nf and output docs need to be changed accordingly.

Allow for index files with just UMI and index files with UMI and sample index

EstimateError can be used to estimate sequencing and PCR error rates: https://presto.readthedocs.io/en/stable/examples/tasks.html#estimating-sequencing-and-pcr-error-rates-with-umi-data

A process could be included to estimate these errors per sample

Description of feature

As of now the input validation of the fastq samplesheet is done with a custom python script. It would be nice that this input would also be validated with the enchantr validate_input script.

Fastq tsv file

Reveal tsv file

The main difference is the column filename which is not present in the fastq tsv as 3 files might be provided in the columns filename_R1, filename_R2, filename_I1. Would it be possible to accept more than filename_xx column and allow also .fq-gz, and .fastq.gz extensions there?

Clonal analysis results should produce consistently both png and svg plots.

New release introduced an issue that graphml clones are not stored properly. Problem in calling dnapars from R script clonal_analysis.R

Include other define clones algorithms as defined here:

https://changeo.readthedocs.io/en/version-0.3.7---igblast-1.7-fix/tools/DefineClones.html

This would give us quite some more fancy features without compromising the current setup 👍

https://nf-co.re/developers/sync

Currently the database download is a process inside the nexflow pipeline but it is actually also included in the dockerfile as part of container build.

We would need to decide which one is the best option and just go for one of them.

Abundance and diversity graphs apper small in the define clones report

• Include all parameters in documentation
• Input file description
• Better usage description
• Short description about RNAseq experimental design

https://github.com/nf-core/bcellmagic/blob/11d19bc2c68633cca93c522736552691fff459da/Singularity#L3

As per discussion on slack, this file is not needed and should be removed from the repo.

The script part in process CHANGEO_MAKEDB can be simplified by providing a folder path to -r, not all individual fasta files:
-r ${imgt_base}/${params.species}/vdj/. This would also fix a bug where Ig light chain reference germlines (IGKV, IGKJ, IGLV, IGLJ) are not being passed to -r.

The archive of IMGT databases can be found at: http://www.imgt.org/download/V-QUEST/archives/

• Need to add minimal tests for travis in test.config.
• Use subset of fastq file for one sample.
• Need to remove real primer sequences for fakes.

Description of the bug

The instructions to run airrflow 2.3.0 according the "Takara Bio SMARTer Human TCR v2" section at https://nf-co.re/airrflow/2.3.0/usage#dt-oligo-rt-and-5race-pcr are as follows:

nextflow run nf-core/airrflow -profile docker \
--input samplesheet.tsv \
--library_generation_method dt_5p_race_umi \
--cprimers CPrimers.fasta \
--race_linker linker.fasta \
--umi_length 12 \
--umi_position R2 \
--cprimer_start 5 \
--cprimer_position R1 \
--outdir ./results

However, when using these settings on my data, the pipeline rapidly exits with the following error:

Cannot invoke method and() on null object

 -- Check script '/home/ubuntu/.nextflow/assets/nf-core/airrflow/./workflows/bcellmagic.nf' at line: 100 or see '.nextflow.log' file for more details

Examining line 100 in bcellmagic.nf, as suggested, gives the following:

if (params.index_file & params.umi_position == 'R2') {exit 1, "Please do not set `--umi_position` option if index file with UMIs is provided."}

Setting the parameter --index_file FALSE solves this issue and the pipeline successfully launches. This suggests the pipeline expects a default value of FALSE to be present for params.index_file, but that this parameter value is in fact empty.

Command used and terminal output

No response

Relevant files

No response

System information

No response

Currently when a threshold could not be defined in the shazam step (naive samples) the pipeline breaks. Solution is to submit it with a manually defined threshold. This should be handled automatically.

Use schema build to generate parameter schema for the pipeline

As reported by @fabio-t , the pipeline currently expects C-region primers to be in the R1 fastq and V-region primers to be in the R2 fastq.

Better docs about this or the possibility of specifying R1/R2 for both primers would be needed.

Hi, this is kojix2.

I'm a newbie and have very little understanding of what bcellmagic is a workflow for.
First, to understand what bcellmagic can do, I ran -profile test,docker in Nextflow Tower and got the following error.

What are the possible causes of this?

 Workflow execution completed unsuccessfully

The exit status of the task that caused the workflow execution to fail was: 1

Error executing process > 'NFCORE_BCELLMAGIC:BCELLMAGIC:ALAKAZAM_SHAZAM_REPERTOIRES (report)'

Caused by:
  Essential container in task exited

Command executed:

  execute_report.R repertoire_comparison.Rmd
  Rscript -e "library(alakazam); write(x=as.character(packageVersion('alakazam')), file='alakazam.version.txt')"
  Rscript -e "library(shazam); write(x=as.character(packageVersion('shazam')), file='shazam.version.txt')"
  echo $(R --version 2>&1) | awk -F' '  '{print $3}' > R.version.txt

Command exit status:
  1

Command output:
    |............................................................          |  86%
    ordinary text without R code
  
  
    |                                                                            
    |..............................................................        |  88%
  label: unnamed-chunk-10 (with options) 
  List of 4
   $ echo     : logi FALSE
   $ fig.width: num 10
   $ fig.asp  : num 0.8
   $ fig.align: chr "center"
  
  
    |                                                                            
    |...............................................................       |  91%
    ordinary text without R code
  
  
    |                                                                            
    |.................................................................     |  93%
  label: unnamed-chunk-11 (with options) 
  List of 4
   $ echo     : logi FALSE
   $ fig.width: num 10
   $ fig.asp  : num 0.8
   $ fig.align: chr "center"
  
  
    |                                                                            
    |...................................................................   |  95%
    ordinary text without R code
  
  
    |                                                                            
    |....................................................................  |  98%
  label: unnamed-chunk-12 (with options) 
  List of 4
   $ echo     : logi FALSE
   $ fig.width: num 10
   $ fig.asp  : num 0.8
   $ fig.align: chr "center"
  
  
    |                                                                            
    |......................................................................| 100%
    ordinary text without R code
  
  
  /usr/local/bin/pandoc +RTS -K512m -RTS repertoire_comparison.knit.md --to html4 --from markdown+autolink_bare_uris+tex_math_single_backslash --output /tmp/nxf.XXXXj9Glzr/Bcellmagic_report.html --lua-filter /usr/local/lib/R/library/rmarkdown/rmarkdown/lua/pagebreak.lua --lua-filter /usr/local/lib/R/library/rmarkdown/rmarkdown/lua/latex-div.lua --self-contained --variable bs3=TRUE --standalone --section-divs --table-of-contents --toc-depth 3 --variable toc_float=1 --variable toc_selectors=h1,h2,h3 --variable toc_collapsed=1 --variable toc_smooth_scroll=1 --variable toc_print=1 --template /usr/local/lib/R/library/rmarkdown/rmd/h/default.html --highlight-style pygments --variable theme=bootstrap --css nf-core_style.css --include-in-header /tmp/Rtmp9ZwV8C/rmarkdown-str3a595dbe04.html --mathjax --variable 'mathjax-url:https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML' --citeproc 

Command error:
  
  
  processing file: repertoire_comparison.Rmd
  output file: repertoire_comparison.knit.md
  
  File ./references.bibtex not found in resource path
  Error: pandoc document conversion failed with error 99
  Execution halted

Work dir:
  /fusion/s3/ABCDEFG/bcellmagic/wd/25/96cdb23bc7f02ad73b2a5a2e087162

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

E.g. using switch and cases groovy functions as part of Workflow.groovy

Original bug report from Susanna Marquez:

I have just now started trying bcellmagic. I get the error below. Any suggestions? .command.sh simply has the same error message, about moving the files.

[14/b08493] process > get_software_versions      [100%] 1 of 1 ✔ 
[-        ] process > multiqc                    - 
[a9/753701] process > output_documentation (1)   [100%] 1 of 1 ✔ 
Execution cancelled -- Finishing pending tasks before exit 
[0;35m[nf-core/bcellmagic] Pipeline completed with errors
Error executing process > 'merge_r1_umi (HD07M)'

Caused by:
 Process `merge_r1_umi (HD07M)` terminated with an error exit status (1)

Command executed:

 gunzip -f "HD07M_R1.fastq.gz"
 mv "HD07M_R1.fastq" "HD07M_R1.fastq"
 gunzip -f "HD07M_R2.fastq.gz"
 mv "HD07M_R2.fastq" "HD07M_R2.fastq"

Command exit status:
 1

Command output:
 (empty)

Command error:
 mv: 'HD07M_R1.fastq' and 'HD07M_R1.fastq' are the same file

Work dir:
 /home/susanna/Documents/projects/collaborations/bcell-magic/work/e3/e7655ee3723008a802d5b887ea585a

• make sure all processes have resource labels
• make sure all processes print the versions properly
• define clones and create germlines results within changeo, not shazam

At the moment, most tools that are build in the docker image do not make use of any conda channel. The aim is to bring all tools in the docker image to conda. In the following, all tools in the container are listed. The bracket notation tells, if the tool is already in conda or not.

• protocol data, utility scripts, pipelines [--> not needed for now]
• muscle muscle
• vsearch vsearch
• cd-hit cd-hit
• blastx + executables blast
• igblast igblast
• phylib phylib
• tbl2asn tbl2asn
• airr reference libraries airr r-airr
• presto presto
• changeo changeo
• alakazam alakazam
• shazam shazam
• tigger tigger
• rdi [--> not needed for now]
• scope [not needed for now]
• prestor [not needed for now]
• download and build reference databases [inside of the container?! --> not]

Description of feature

It would be nice to report the primer / linker sequences in the airrflow report for reproducibility.

As proposed by @ssnn-airr, it would be nice to separate the process in fetch_databases between pulling the references from IMGT and building the igblast references, to also allow users to provide their own fasta databases but still build them with igblast.

Currently, the fetch_databases.sh script is executed in the Fetch_databases process which calls the underlying scripts:

• fetch_imgt.sh pulls the references directly from IMGT for 4 species and stores them in fasta format.
• fetch_igblastdb.sh gets some internal data needed for igblast
• imgt2igblast.sh uses makeblastdb to create the necessary Igblast references.

That could be separated into two processes, one for pulling the fasta references, and one for building the necessary references for igblast.

So if the reference data is available in IMGT, one could adapt the first script to also pull it and then build the igblast needed references. Or if custom reference data is available in fasta format, it should also be possible to provide it directly to imgt2igblast to build the references.

Hi!

this is not necessarily an issue with the pipeline, but in order to streamline the documentation group next week for the hackathon, I'm opening issues in all repositories / pipeline repos that might need this update to switch from parameter docs to auto-generated documentation based on the JSON schema.

This will then supersede any further parameter documentation, thus making things a bit easier :-)

If this doesn't apply (anymore), please close the issue. Otherwise, I'm hoping to have some helping hands on this next week in the documentation team on Slack https://nfcore.slack.com/archives/C01QPMKBYNR

Add configs for process labels and use them in processes

It would be nice if the pipeline didn't crash when using plain FASTQ files, rather than gzipped ones :)

Not a huge deal, but it's not specified into the documentation (or I missed it).

Adding an output table with all numbers also in the report file size report would be useful as a summary.

It could be useful to be able to use bcellmagic on datasets without UMIs.

I was thinking of using the Greiff2014_Workflow data subset from the presto docs as a minimal/test example for development.

• Join samples belonging to the same patient before running define clones.
• Check join operator from nextflow to do this.

Hi,
I'm looking at all pipeline, and I noticed that the dev branch has not been bumped to dev.

Description of feature

Updating Immcantation tool versions for mulled containers (here the packages with the old versions are listed):

• conda-forge::r-base=4.1.2 bioconda::r-alakazam=1.2.0 bioconda::changeo=1.2.0 bioconda::phylip=3.697 conda-forge::r-optparse=1.7.1
• conda-forge::r-base=4.1.2 bioconda::r-alakazam=1.2.0 bioconda::r-shazam=1.1.0 conda-forge::r-kableextra=1.3.4 conda-forge::r-knitr=1.33 conda-forge::r-stringr=1.4.0 conda-forge::r-dplyr=1.0.6 conda-forge::r-optparse=1.7.1
• bioconda::changeo=1.2.0 bioconda::igblast=1.17.1 conda-forge::wget=1.20.1
• conda-forge::r-base=4.1.2 bioconda:r-alakazam=1.2.0 bioconda::changeo=1.2.0 bioconda::igphyml=1.1.3
• bioconda::changeo=1.2.0 bioconda::igblast=1.17.1
• conda-forge::r-base=4.1.2 bioconda:r-enchantr=0.0.1
• enchantr
• mulled
  And whatever containers are needed for the reveal processes :D