ctmrbio / bactpipe Goto Github PK
View Code? Open in Web Editor NEWBACTpipe: An assembly and annotation pipeline for bacterial genomics
Home Page: https://bactpipe.readthedocs.org
License: MIT License
BACTpipe: An assembly and annotation pipeline for bacterial genomics
Home Page: https://bactpipe.readthedocs.org
License: MIT License
I suggest we consider putting all user-configureable tool parameters (e.g. bbduk trimming settings, assembly kmer sizes, maybe some prokka parameters, etc) in a single configuration file. Having all of these settings in a single file makes it easier to properly maintain, and also easier for end-users to access and modify parameters as desired, without having to go into the actual code for the main Nextflow workflow.
There are a few models for how to do this, but I suggest @b16joski takes a look at a few other Nextflow-based pipelines to see how the manage hierarchical configuration files and profiles. (e.g. https://github.com/ewels/NGI-RNAseq could be a starting point, but I don't think their solution is exactly what we are after).
We can have one output file summarizing mash screen results for the users. Also we can remove any intermediate files from the various BACTpipe output directories.
And don't forget to make mini-celebration when this is done :)
There's an easy way to ensure Nextflow doesn't include the input reads in the mash screen output folder.
Change this line: https://github.com/ctmrbio/BACTpipe/blob/master/bactpipe.nf#L21
publishDir "${params.output_dir}/mash.screen", mode: 'copy'
to
publishDir "${params.output_dir}/mash.screen", mode: 'copy', pattern: '*.{txt,tsv}'
This needs to be fixed in some way for the future.
Rewrite in Python 3.5+ and preferably remove unnecessary Biopython dependency.
It felt wrong having to always specify the path to adapters.fa
for BBDuk, and I did some research and obviously it is possible for BBDuk to use the adapters.fa
file that comes bundled with BBMap when installing it, without having to specify the entire path. It's dead easy: adapters
is a keyword that BBDuk knows means the adapters file bundled with the installation. We can just give BBDuk ref=adapters
instead of ref=/path/to/adapters.fa
and it will automatically use the adapters file that is bundled. This removes the need to automatically download it if no path for a custom adapters.fa
is given by the user, and cleans up a lot of the messy code we recently introduced into bactpipe.nf
to handle all the edge cases. I really think we should consider fixing this before releasing BACTpipe publicly.
I suggest we create a branch to hold the last version of BACTpipe version1 in the repo. I don't expect any of us are interested in maintaining it now that we've improved so much in version 2, so let's just keep it as an archive in a side branch.
We could easily do this before merging the PR from Joseph's version 2 branch.
This is the output file on the Brachyspira aalborgi PC4226IV PC4226IV.mash_screen.txt
I must have accidentally removed params.project
from the configuration files when I restructured everything a while back. Currently, you need to specify --project <SLURMPROJECT>
when running BACTpipe on Rackham or Milou.
I suggest we add params.project = ""
to the default config file (params.config
), and add some logic to the main workflow to show a warning and exit gracefully if it has not been set properly.
When running BACTpipe v2.0 with Yue just now, we noticed an unfortunate syntax error in nextflow.config
.
https://github.com/ctmrbio/BACTpipe/blob/master/nextflow.config#L17
There is no closing bracket for the rackham profile definition.
We need to determine the best way to identify a reference species based on the assembled sequences (contigs.fa
). Current ideas have touched upon MASH, but also Jspecies, etc. Still no solution.
specify if wanted by the user for prokka to use specific reference sequences during annotation.
Replace with
Parameter 'mashscreen_database' is empty. Download from https://address-to-database.
So that instead of that the pipeline outputs all prokka output files into the same folder, there will be separate folders, i.e. prokka/${sample_id}
I missed that Brian Bushnell last year added a tool to the BBTools suite that does something extremely similar to what mash screen
does: sendsketch.sh
.
It sends a sketch of an input file (or pair of input files, compressed or not, as per usual BBTool manners) to JGI's sketch-server to compare against reference sketches of nt
, refseq
(default), silva
, or img
. It is very fast. Here's an example using the first 1000 or so reads from a Helicobacter pylori sample:
(base) [fredrik.boulund@ctmr-nas bactpipe_test]$ sendsketch.sh in=sample_R1.fastq.gz
Adding /home/ctmr/anaconda3/opt/bbmap-37.90/resources/blacklist_refseq_species_300.sketch to blacklist.
Loaded 1 sketch in 0.716 seconds.
Query: HWI-M03284:45:000000000-AWVDC:1:1101:13827:1954 1:N:0:GAATTCGTGTACTGAC DB: RefSeq SketchLen: 8587 Seqs: 26873 Bases: 8088773 gSize: 2134365 Quality: 0.9074 File: sample_R1.fastq.gz
WKID KID ANI Complt Contam Matches Unique noHit TaxID gSize gSeqs taxName
36.19% 30.08% 96.37% 54.43% 34.52% 2752 0 2227 102617 2560243 63 Helicobacter pylori SS1
7.66% 5.46% 91.08% 47.88% 65.86% 469 3 2463 382638 1530316 2 Helicobacter acinonychis str. Sheeba
0.95% 0.77% 84.42% 36.38% 70.55% 66 0 2463 1163745 1730296 2 Helicobacter cetorum MIT 99-5656
0.79% 0.55% 83.87% 42.33% 70.77% 47 0 2463 104628 1475704 42 Helicobacter suis
3.03% 0.03% 88.06% 100.00% 71.28% 3 0 2463 1204178 26419 1 Helicobacter phage KHP40
0.31% 0.07% 81.06% 6.77% 70.61% 10 0 937 543736 9071270 292 Rhodococcus opacus PD630
0.12% 0.12% 78.34% 30.08% 71.20% 10 0 2463 1578720 2046283 204 Helicobacter ailurogastricus
0.12% 0.09% 78.37% 37.89% 71.22% 8 0 2463 936155 1631038 1 Helicobacter felis ATCC 49179
0.11% 0.09% 78.10% 22.32% 70.79% 8 0 2087 35817 2772762 392 Helicobacter heilmannii
0.09% 0.07% 77.36% 35.40% 71.25% 6 0 2463 1002805 1729718 147 Helicobacter bizzozeronii CCUG 35545
0.07% 0.06% 76.61% 32.44% 71.26% 5 0 2463 537972 1901982 44 Helicobacter pullorum MIT 98-5489
0.06% 0.05% 76.58% 40.04% 71.27% 4 0 2463 679897 1539411 1 Helicobacter mustelae 12198
0.06% 0.05% 76.43% 37.98% 71.27% 4 0 2463 537970 1610516 24 Helicobacter canadensis MIT 98-5491
0.06% 0.05% 76.33% 36.61% 71.27% 4 0 2463 556267 1653215 21 Helicobacter winghamensis ATCC BAA-430
0.06% 0.05% 76.16% 34.44% 71.27% 4 0 2463 1449345 1781940 29 Helicobacter rodentium ATCC 700285
0.05% 0.05% 76.06% 33.23% 71.27% 4 0 2463 1476199 1851996 61 Helicobacter sp. 12S02634-8
0.05% 0.05% 75.71% 29.32% 71.27% 4 0 2463 1325130 2093688 49 Helicobacter fennelliae MRY12-0050
0.05% 0.05% 75.72% 27.56% 71.39% 4 0 2398 211 2212464 224 Helicobacter sp. CLO-3
0.05% 0.03% 75.99% 43.25% 71.28% 3 0 2463 1408442 1419428 11 Helicobacter pametensis ATCC 51478
0.04% 0.03% 75.39% 34.70% 71.28% 3 0 2463 1905759 1770278 45 Helicobacter sp. 13S00477-4
Total Time: 2.198 seconds.
As you can see, it works very well! Of course, it would require some tweaking of assess_mash_screen.py
, to parse sendsketch.sh
output instead, and some additional testing like the testing and validation we performed for our mash screen
evaluations, but it shouldn't be too much work to be honest.
We could consider eventually replacing mash screen
with sendsketch.sh
, thus removing the entire mash dependency, and removing the need to download a 700MB+ file with sketches of RefSeq genomes.
edit: Here's Brian Bushnell's "announcement" of sendsketsh.sh
on BioStars: https://www.biostars.org/p/234837/
...and make sure it is python 3-compatible
At minimum: create docs folder, and Sphinx skeleton.
As a stretch-goal for the next main version of the workflow, I'd like us to implement automated regression testing. It's not very difficult, and will improve our ability to catch small bugs, typos, etc. in pull requests and commits as early as possible.
In order to implement this in a useful way, we need at least one test case for each major path through the workflow. E.g. one gram positive, one gram negative, one with reference proteins, one which is contaminated, etc., so we can validate that all common paths through the workflow works as they should.
I'm labeling this issue as low priority and assigning it to the BACTpipe v3.0 milestone for now. There's no rush to implement this at this time.
For version 2.0, we need a flowchart showing the branching workflow.
I propose we transfer ownership of the bactpipe_nextflow
repository to the CTMR github organization when it reaches usable maturity. I also propose we change the repository name to just bactpipe
when transferred to the CTMR organization.
It won't change anything for access of ease-of-use for us as developers, but it keeps it nicely under the CTMR organization umbrella. After such a transfer, it is still possible for all involved authors to fork the new main repository into their personal github accounts. This also improves the development workflow, as it enables individual researchers to develop code improvements in their local forks, which can the be transferred to the main repo via pull requests. This has the added benefit of making it easier for all involved parties to review all incoming changes.
Think about it, so we can discuss at a later time.
To be reintroduced if someone has a reason and time to write a tutorial.
We need to clarify in the documentation that users of the pipeline have to have progressiveMauve and t2blasn in their PATH or in the bactpipe bin.
...and if so, make bbduk also use the phiX genome file as well
There are currently a few kinks to work out regarding automatic downloading of reference databases (BBDuk adapters, mash screen db).
I'll continue work on this asap.
There's little reason to run BBMap's stats.sh
(or if we run statswrapper.sh
, I'm not sure) as its own process, as it leads to unnecessary file copying/staging for a very little process. I think it's better to move it into the assembly step, and run it after shovill completes inside the same process.
I propose changes to the main folder of the repo to better follow Nextflow best practices of repo organization. We will create a separate configuration folder, and introduce some new files to make it easier to run the workflow directly from the terminal without downloading it beforehand using Nextflow's built-in functionality for running workflows directly out of Github repos.
... and the other programs to use the gzipped files as input files
I suggest to only output the draft genome in the form of the prokka .fna file and not also the contig.fa file from Shoville since these will be basically identical.
I just noticed that there are some large commits that I think might have been accidental. Amongst other things, the commit in question includes the source code for another program (FastML), as well as a compiled version. This is a severe problem. We can't have the source code and binaries (or source code tarballs) of other software commited into the repo. Firstly, because it in almost all cases violates the copyright and distribution licenses of these tools, secondly because it clutters and unnecessarily bloats the repository, and thirdly because it would give us a lot more work keeping these third-party tools up-to-date in our repository (if we're even allowed to include them in the first place).
Instead, I think we should put links to the websites of all important dependencies in the README so that pipeline users easily can find and download the prerequisite software. That way, we don't have to take responsibility to keep the latest version of these tools inside our own repository either. Just make sure to include the version of each tool that has been tested with the pipeline.
@b16joski, this is an excellent opportunity for you to learn how to remove accidentally commited files from a git repository. It is not enough to just remove them in a new commit, since they will still remain in the repository commit history. They need to be entirely removed from history as well. I've only done this kind of operation once, and that was in Mercurial, not Git.
As far as I know, there are only two reasonable options: git filter-branch
or https://rtyley.github.io/bfg-repo-cleaner/ . Research what is the best option in this case and remove all the third-party files from the repository history.
Make sure you document your steps to solving this issue in this discussion thread so we can all refer to it in the future. Let us know if you encounter issues and need help. This is a tricky operation that could possibly screw up the repository (changing history is very precarious).
Show all steps in the "one path" version of the pipeline, similar to the draft in Joseph's thesis.
I will soon submit a pull request for changes to unify the coding style throughout the (nextflow) code base. I won't touch any of the third-party scripts.
Add multiQC step at the end of the pipeline workflow
Reduce allocations for:
So that the fasta headers in the clean file that are used as input to prokka are of the format
${sample_id}_contig1
Containerization is a technique to make it easier for end users to run stuff without having to install a lot of complicated dependencies. Common tools for this task are Docker and Singularity. Using containers is also very convenient when running on distributed cluster resources, as the container is a complete package containing all of the dependencies (libraries, scripts, tools, etc) required to run the different pipeline steps.
For our case with BACTpipe, containerization would mean that users only need to have Docker and Nextflow installed in their environment in order to run all of BACTpipe, without having to mess with installing any of the tools used inside the BACTpipe workflow (in theory, users would even be able to run BACTpipe on a Windows machine without too much trouble). Nextflow comes with Docker support out of the box, and it is very easy to use.
A simple approach to containerization of BACTpipe would be to just create a single container (e.g. a Docker image) that contains a miniature Linux environment with all of the tools used by BACTpipe. As Nextflow has built-in support, it is really quite easy to make a BACTpipe container and make Nextflow use that when executing the different processes in the BACTpipe workflow.
I already made an image for BACTpipe version 2.2-dev that works just fine. I recently pushed a new branch, docker_test
, that makes some tiny changes to the Nextflow configuration to make it run the workflow processes inside Docker containers based on the Docker image I made. Nextflow still runs on your machine, but all the processes inside the workflow run inside instances of the Docker container.
Despite it working very well for now, there are still some improvements to be made:
Github has features to create "releases" using tagged commits. Now that we have an old version of the pipeline (version 1) available in a separate branch, it is easy to create a tagged release for it. Would you read up on how to do that @b16joski? Ask me if you get stuck.
Tried running BACTpipe on 17 PhiX-contaminated Brachyspira genomes, both on Rackham and Milou clusters. See log files below.
So that one can run it either assuming all dependencies are in the $PATH or, if one is working in milou/other server can call the dependencies from modules.
Pyani is good example here. One with good visual outputs or publication ready figures could be very nice here!!! This could act as also be a confirmatory test for mash screen predictions.
To gather assembly stats.
Update and improve the pipeline documentation
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.