nf-core / cookiecutter Goto Github PK

View Code? Open in Web Editor NEW

19.0 19.0 8.0 153 KB

DEPRECIATED! Please use nf-core/tools instead

Home Page: http://nf-co.re

License: MIT License

HTML 7.61% R 3.35% Python 4.00% Nextflow 84.03% Dockerfile 1.01%

bioinformatics bioinformatics-pipeline cookiecutter cookiecutter-template nextflow pipeline template workflow

cookiecutter's People

Contributors

Stargazers

Watchers

Forkers

ewels sven1103 apeltzer remiolsen alneberg sofiahag urish biofriends

cookiecutter's Issues

Flag ignored processes at end of run

Normally, ignored tasks are a bad thing in nf-core pipelines. It would be good to add some template code that flags a big warning when pipelines finish if any tasks errored and were ignored.

Better handling of container addresses

Singularity and docker container addresses are currently not handled in the best way. This is made difficult by the fact that we have nfcore on dockerhub and nf-core on singularity hub 🤦‍♂️

This issue continues discussion started on PR #42

Speed up Conda Resolving

To my knowledge, conda package resolving should speed up significantly, when we specify the way packages are pulled by a corresponding prefix, e.g.:

- conda-forge::jdk=1.X.Y

is much faster than
- jdk=1.X.Y

Maybe we should make this something that is automatically added to the environment.yml when a new repository is created to make users more aware of this?

Update to new withName syntax

We should be using the new withName syntax in the configuration files, instead of the older process$name syntax.

Conda channels are reversed

See bioconda/bioconda-recipes#8588 (comment)

Should be:

channels:
  - bioconda
  - conda-forge
  - defaults

Merge into tools repo

It's possible to use cookiecutter from within python, in addition to as a command line tool: https://github.com/audreyr/cookiecutter

This means that we could easily make cookiecutter a python dependency in the tools package, include the cookiecutter template files in that repository and then have a subcommand to create a pipeline, eg. nf-core create_pipeline or something

Merging the two repos will mean that the linting and templates are kept in sync and that the main nf-core command does everything (easier for end users).

Figure out how to handle singularity mount points for many users

One of the biggest problems that I can see with singularity images in pipelines designed to be used by everyone is bind points - mounting root directories within the container. The ENABLE_OVERLAYS option doesn't seem to be set for most people, so currently all root directory paths have to be added in the docker build script. This isn't exactly scaleable and we could do with a better solution.

Is there a better way to handle this?

Add support for bioconda built containers

Write a docker build script that installs all dependencies using a bioconda env file. Then we can use bioconda, docker and singularity all from a single script which is super easy to maintain 🎉

Add TODO statements in template

Dear all,

just started to port a pipeline into nf-core, and realized, that it might me a cool addition to have //TODO <description> comment lines in the cookiecutter template, for example for the help message function you need to adapt, and so on.

Then you could easily display them in your favourite IDE and don't forget something to implement/change. Moreover, we could also check on a release with nf-core/tools, that all //TODO tags are removed in the code?
Just brainstorming here, happy for your feedback!

Example

Have a tag //TODO nf-core:.

Were?:

main.nf
- helpMessage()
- log info
Dockerfile
Singularity
nextflow config
...

Best, Sven

Don't use executor local

Forcing executor: local for specific processes can break the pipeline for some systems (see nextflow-io/nextflow#693 (comment)).

Update this and all nf-core pipelines to remove that.

Improve documentation

The docs are a little sparse in a few places, and some other pipelines have better, more comprehensive stuff written already (such as the methylseq pipeline). Would be good to go through all pipelines and homogenise the docs across all of them.

Recommend setting singularity cache directory

We should instruct people to set NXF_SINGULARITY_CACHEDIR as part of the installation instructions.

Sort out better release tagging for singularity

Docker hub works nicely with automated builds based on GitHub releases which are tagged with the same name.

Singularity hub doesn't support the same method and instead has nasty filename based tagging support. However, there is a very nice CLI that interacts with the API.

We should be able to use this CLI to automatically build and tag singularity hub containers when GitHub releases are prepared, using Travis CI.

Build MIT license from template

Currently, we enforce the provision of an MIT license, but the cookiecutter template does not create one by default.

bioconda docker problem - no fonts found

We may have a slight problem with the base docker container that we're using for the bioconda builds. I just tried to run FastQC and got the following log:

Exception in thread "Thread-1" java.lang.Error: Probable fatal error:No fonts found.
	at sun.font.SunFontManager.getDefaultPhysicalFont(SunFontManager.java:1236)
	at sun.font.SunFontManager.initialiseDeferredFont(SunFontManager.java:1100)
	at sun.font.SunFontManager.findOtherDeferredFont(SunFontManager.java:1037)
	at sun.font.SunFontManager.findDeferredFont(SunFontManager.java:1054)
	at sun.font.SunFontManager.findFont2D(SunFontManager.java:2256)
	at sun.font.SunFontManager.getDefaultPhysicalFont(SunFontManager.java:1220)
	at sun.font.SunFontManager.initialiseDeferredFont(SunFontManager.java:1100)
	at sun.font.SunFontManager.findOtherDeferredFont(SunFontManager.java:1037)
	at sun.font.SunFontManager.findDeferredFont(SunFontManager.java:1054)
	at sun.font.SunFontManager.findFont2D(SunFontManager.java:2256)
	at sun.font.SunFontManager.getDefaultPhysicalFont(SunFontManager.java:1220)
	at sun.font.SunFontManager.initialiseDeferredFont(SunFontManager.java:1100)
	at sun.font.SunFontManager.findOtherDeferredFont(SunFontManager.java:1037)
	at sun.font.SunFontManager.findDeferredFont(SunFontManager.java:1054)
	at sun.font.SunFontManager.findFont2D(SunFontManager.java:2256)
	at sun.font.SunFontManager.getDefaultPhysicalFont(SunFontManager.java:1220)
	at sun.font.SunFontManager.initialiseDeferredFont(SunFontManager.java:1100)
	at sun.font.SunFontManager.findOtherDeferredFont(SunFontManager.java:1037)
	at sun.font.SunFontManager.findDeferredFont(SunFontManager.java:1054)
	at sun.font.SunFontManager.findFont2D(SunFontManager.java:2256)
	at sun.font.SunFontManager.getDefaultPhysicalFont(SunFontManager.java:1220)
	at sun.font.SunFontManager.initialiseDeferredFont(SunFontManager.java:1100)
	at sun.font.SunFontManager.findOtherDeferredFont(SunFontManager.java:1037)
	at sun.font.SunFontManager.findDeferredFont(SunFontManager.java:1054)
	at sun.font.SunFontManager.findFont2D(SunFontManager.java:2256)
	at sun.font.SunFontManager.getDefaultPhysicalFont(SunFontManager.java:1220)
	at sun.font.SunFontManager.initialiseDeferredFont(SunFontManager.java:1100)
	at sun.font.SunFontManager.findOtherDeferredFont(SunFontManager.java:1037)
	at sun.font.SunFontManager.findDeferredFont(SunFontManager.java:1054)
	at sun.font.SunFontManager.findFont2D(SunFontManager.java:2256)
	at sun.font.SunFontManager.getDefaultPhysicalFont(SunFontManager.java:1220)
	at sun.font.SunFontManager.initialiseDeferredFont(SunFontManager.java:1100)
	at sun.font.SunFontManager.findOtherDeferredFont(SunFontManager.java:1037)
	at sun.font.SunFontManager.findDeferredFont(SunFontManager.java:1054)
	at sun.font.SunFontManager.findFont2D(SunFontManager.java:2256)
	at sun.font.SunFontManager.getDefaultPhysicalFont(SunFontManager.java:1220)
	at sun.font.SunFontManager.initialiseDeferredFont(SunFontManager.java:1100)
	at sun.font.SunFontManager.findOtherDeferredFont(SunFontManager.java:1037)
	at sun.font.SunFontManager.findDeferredFont(SunFontManager.java:1054)
	at sun.font.SunFontManager.findFont2D(SunFontManager.java:2256)
	at sun.font.SunFontManager.getDefaultPhysicalFont(SunFontManager.java:1220)
	at sun.font.SunFontManager.initialiseDeferredFont(SunFontManager.java:1100)
	at sun.font.SunFontManager.findOtherDeferredFont(SunFontManager.java:1037)
	at sun.font.SunFontManager.findDeferredFont(SunFontManager.java:1054)
	at sun.font.SunFontManager.findFont2D(SunFontManager.java:2256)
	at sun.font.SunFontManager.getDefaultPhysicalFont(SunFontManager.java:1220)
	at sun.font.SunFontManager.initialiseDeferredFont(SunFontManager.java:1100)
	at sun.font.SunFontManager.findOtherDeferredFont(SunFontManager.java:1037)
	at sun.font.SunFontManager.findDeferredFont(SunFontManager.java:1054)
	at sun.font.SunFontManager.findFont2D(SunFontManager.java:2256)
	at sun.font.SunFontManager.getDefaultPhysicalFont(SunFontManager.java:1220)
	at sun.font.SunFontManager.initialiseDeferredFont(SunFontManager.java:1100)
	at sun.font.SunFontManager.findOtherDeferredFont(SunFontManager.java:1037)
	at sun.font.SunFontManager.findDeferredFont(SunFontManager.java:1054)
	at sun.font.SunFontManager.findFont2D(SunFontManager.java:2256)
	at sun.font.SunFontManager.getDefaultPhysicalFont(SunFontManager.java:1220)
	at sun.font.SunFontManager.initialiseDeferredFont(SunFontManager.java:1100)
	at sun.font.SunFontManager.findOtherDeferredFont(SunFontManager.java:1037)
	at sun.font.SunFontManager.findDeferredFont(SunFontManager.java:1054)
	at sun.font.SunFontManager.findFont2D(SunFontManager.java:2256)
	at sun.font.SunFontManager.getDefaultPhysicalFont(SunFontManager.java:1220)
	at sun.font.SunFontManager.initialiseDeferredFont(SunFontManager.java:1100)
	at sun.font.SunFontManager.findOtherDeferredFont(SunFontManager.java:1037)
	at sun.font.SunFontManager.findDeferredFont(SunFontManager.java:1054)
	at sun.font.SunFontManager.findFont2D(SunFontManager.java:2256)
	at sun.font.SunFontManager.getDefaultPhysicalFont(SunFontManager.java:1220)
	at sun.font.SunFontManager.initialiseDeferredFont(SunFontManager.java:1100)
	at sun.font.SunFontManager.findOtherDeferredFont(SunFontManager.java:1037)
	at sun.font.SunFontManager.findDeferredFont(SunFontManager.java:1054)
	at sun.font.SunFontManager.findFont2D(SunFontManager.java:2256)
	at sun.font.SunFontManager.getDefaultPhysicalFont(SunFontManager.java:1220)
	at sun.font.SunFontManager.initialiseDeferredFont(SunFontManager.java:1100)
	at sun.font.SunFontManager.findOtherDeferredFont(SunFontManager.java:1037)
	at sun.font.SunFontManager.findDeferredFont(SunFontManager.java:1054)
	at sun.font.SunFontManager.findFont2D(SunFontManager.java:2256)
	at sun.font.SunFontManager.getDefaultPhysicalFont(SunFontManager.java:1220)
	at sun.font.SunFontManager.initialiseDeferredFont(SunFontManager.java:1100)
	at sun.font.SunFontManager.findOtherDeferredFont(SunFontManager.java:1037)
	at sun.font.SunFontManager.findDeferredFont(SunFontManager.java:1054)
	at sun.font.SunFontManager.findFont2D(SunFontManager.java:2256)
	at sun.font.SunFontManager.getDefaultPhysicalFont(SunFontManager.java:1220)
	at sun.font.SunFontManager.initialiseDeferredFont(SunFontManager.java:1100)
	at sun.font.SunFontManager.findOtherDeferredFont(SunFontManager.java:1037)
	at sun.font.SunFontManager.findDeferredFont(SunFontManager.java:1054)
	at sun.font.SunFontManager.findFont2D(SunFontManager.java:2256)
	at sun.font.SunFontManager.getDefaultPhysicalFont(SunFontManager.java:1220)
	at sun.font.SunFontManager.initialiseDeferredFont(SunFontManager.java:1100)
	at sun.font.SunFontManager.findOtherDeferredFont(SunFontManager.java:1037)
	at sun.font.SunFontManager.findDeferredFont(SunFontManager.java:1054)
	at sun.font.SunFontManager.findFont2D(SunFontManager.java:2256)
	at sun.font.SunFontManager.getDefaultPhysicalFont(SunFontManager.java:1220)
	at sun.font.SunFontManager.initialiseDeferredFont(SunFontManager.java:1106)
	at sun.font.CompositeFont.doDeferredInitialisation(CompositeFont.java:263)
	at sun.font.CompositeFont.getSlotFont(CompositeFont.java:352)
	at sun.font.CompositeStrike.getStrikeForSlot(CompositeStrike.java:78)
	at sun.font.CompositeStrike.getFontMetrics(CompositeStrike.java:93)
	at sun.font.FontDesignMetrics.initMatrixAndMetrics(FontDesignMetrics.java:359)
	at sun.font.FontDesignMetrics.<init>(FontDesignMetrics.java:350)
	at sun.font.FontDesignMetrics.getMetrics(FontDesignMetrics.java:302)
	at sun.java2d.SunGraphics2D.getFontMetrics(SunGraphics2D.java:855)
	at uk.ac.babraham.FastQC.Graphs.QualityBoxPlot.paint(QualityBoxPlot.java:88)
	at javax.swing.JComponent.print(JComponent.java:1202)
	at uk.ac.babraham.FastQC.Modules.AbstractQCModule.writeDefaultImage(AbstractQCModule.java:68)
	at uk.ac.babraham.FastQC.Modules.PerBaseQualityScores.makeReport(PerBaseQualityScores.java:199)
	at uk.ac.babraham.FastQC.Report.HTMLReportArchive.<init>(HTMLReportArchive.java:131)
	at uk.ac.babraham.FastQC.Analysis.OfflineRunner.analysisComplete(OfflineRunner.java:155)
	at uk.ac.babraham.FastQC.Analysis.AnalysisRunner.run(AnalysisRunner.java:110)
	at java.lang.Thread.run(Thread.java:745)

FastQC then seems to be hanging..

This makes me a little nervous about similar problems that we may have with other packages too if basing off a container so lightweight that it doesn't even have any fonts.

Documentation redundancy

The files local.md and adding_your_own.md have similar (if not identical) sections for Docker and Singularity - we should remove these in one location at least :-)

AWSBatch Configuration

Hi!

I think we could have a generic AWSBatch configuration in cookiecutter.

Did so for ICGC-featureCounts but we could use a similar way (open for ideas if thats not optimal in your opinion):

AWSBatch config:

https://github.com/nf-core/ICGC-featureCounts/blob/master/conf/awsbatch.config

Main Nextflow.config / setting some defaults:
https://github.com/nf-core/ICGC-featureCounts/blob/master/nextflow.config

And then allowing users to specify the required params /also using that in the summary if the proper profile is used:

https://github.com/nf-core/ICGC-featureCounts/blob/4929959b77d13169a8c96ef4abec3ff8b8af314e/main.nf#L138

I guess this could as well be extrapolated to other pipelines easily and could be in cookiecutter therefore!

I'd be happy to contribute that but would like to have some more feedback/ideas on this before moving on...

Describe -offline in documentation

I just came across nextflow-io/nextflow#665 by accident, which I didn't know about. I think this would be useful for a lot of our users (in Sweden on the Bianca cluster especially). Probably mostly by recommending that people export NXF_OFFLINE=true on such environments.

This can be added to the core cookiecutter documentation.

Clean up Conda environment

This should save us a bit space:

SciLifeLab/Sarek#551 (comment)

(wil ldo that later..I think its a good idea to clean up the installers after the installation!)

Add support for conda

Will soon be available in release 0.30.x. See:

https://github.com/nextflow-io/nextflow/blob/master/docs/conda.rst
https://github.com/nextflow-io/nextflow/blob/master/docs/process.rst#conda
https://github.com/nextflow-io/nextflow/blob/master/docs/config.rst#scope-conda

Cookiecutter version should match the nf-core tools version

Idealy, to be consistent, a nf-core tools version should have a matching cookiecutter template version, so we can easily find corresponding pairs :) My suggestion is, that if they have the same version, the linting should pass.

Refactor config to import base.config first, outside of profile scope

See example of where I started doing this on the rnaseq pipeline here: https://github.com/ewels/nf-core-rnaseq/compare/config_refactor#diff-c79fe4336e72c04860afccd21f4ae1c5R17

Execute conditional lint execution in `travis.yml` when on `master` branch

Motivated by the enhancement of nf-core lint nf-core/tools#24.

Adapt the .travis.yml template such that git activity on the master branch executes nf-core lint with the --release flag for production-ready workflows.

Conditionally execute lint tests with --release flag, when git activity happens on the master branch (check env var TRAVIS_BRANCH)

Add the nextflow summary to MultiQC reports

In other pipelines we are adding the summary variables to MultiQC as a config block. See https://github.com/wikiselev/rnaseq/blob/eaebf588e83e2f78cea0a4451db2d4eea5789493/main.nf#L1036-L1054

Add this into the cookiecutter recipe. @pditommaso thinks it "could be replaced by a few lines of groovy" so probably room for some refactoring too 😉

Implement some kind of validation tests

It would be nice to add in some checks for pipeline output files when -profile test is used. This will need to be specific for all pipelines, but we could add an example here.

I think the simplest would be to have a bunch of processes which are downstream of the normal ones and have when: workflow.process.contains('test') set so that they don't run normally.

See extensive discussion on nextflow gitter from here.

Don't use bioconda environments in docker image

See nf-core/chipseq#10 (comment) by @tiagochst:

Changed RUN conda env create -f /environment.yml to RUN conda env update -n root --file environment.yml because of the source below:
- If you are using the Conda package manager, avoid creating a Conda environment. Instead, update the root environment with whatever dependencies you want to install.
  
  As an example to the above warning, it would be best practice to run this: RUN conda env update -n root --file environment.yml and avoid this: RUN conda env create -f environment.yml

https://docs.datascience.com/en/master/appendix-1/dockerfile-basics-and-best-practices-1.html

We should update all pipelines to use this method.