jlanga / exfi Goto Github PK

1. Use version control.

• Put everything that you write or make into version control as soon as it is created.
• Use a feature branch workflow.

2. Document your code and usage.

• [] Write a good README file.
• [] Print usage information.

3. Make common operations easy to control.

• [] Allow the most commonly changed parameters to be configured from the command line.
• [] Check that all input values are in a reasonable range at startup.
• [] Choose reasonable defaults where they exist.
• [] Set no defaults at all when there aren’t any reasonable ones.

4. Version your releases.

• [] Increment your version number every time you release your software to other people.
• [] Make the version of your software easily available by supplying --version or -v on the command line.
• [] Include the version number in in all of the program’s output.
• [] Ensure that old released versions continue to be available.

5. Reuse software (within reason).

• [] Make sure that you really need the auxiliary program.
• [] Ensure the appropriate software and version is available.
• [] Ensure that reused software is robust.

6. Rely on build tools and package managers for installation.

• [] Document all dependencies in a machine-readable form.
• [] Avoid depending on scripts and tools which are not available as packages.

7. Do not require root or other special privileges to install or run.

• [] Do not require root privileges to set up or use packages.
• [] Allow packages to be installed in an arbitrary location.

8. Ask another person to try and build your software before releasing it.

• [] Eliminate hard-coded paths.
• [] Set the names and locations of input and output files as command-line parameters.

9. Do not require users to navigate to a particular directory to do their work.

• [] Include a small test set that can be run to ensure the software is actually working.
• [] Make the tests easy to find and run.
• [] Make the test script’s output easy to interpret.

10. Produce identical results when given identical inputs.

• [] Echo all parameters and software versions to standard out or a log file alongside the results.
• [] Produce the same results each time the same version of the program is run with the same inputs.
• [] Allow the user to optionally provide the random seed as an input parameter.
• [] Make sure acceptable tolerances are known and detailed in documentation and tests.

polish is taking too much memory per thread

biobloommaker returns the files categories_multimatch.fa (empty), categories_noMatch.fa (huge) and categories_transcriptome.fa (not so huge, but big).

These files can be compressed to .gz as they come out by creating a FIFO in their path, and setting gzip in the background.

Or just compress as soon as biobloommaker finishes.

The Bloom filter is written/copied to the .bf file, but the accompanying .txt file that biobloommaker also outputs seems to be absent. However, the BioBloomTools documentation notes that the filter file is useless without the info file. It would be prudent to also keep the info file around after build_baited_bloom_filter.

Bloom filters in this context are mostly zeros.

Using pigz -9 I was able to go from 30Gb to 70Mb in a BF with FPR of 0.01%.

I think also usign gzip instead of pigz will make it even smaller.

Also: try bgzip or xz

By default in the script, only reads are added to the bloom filter.

We could give an option to include both reads in the pair.

exfi build_baited_bloom_filter : build the baited bloom filter from biobloom|abyss
exfi build_splice_graph :
exfi gfa1_to_exons :
exfi gfa1_to_gapped_transcripts :

Sealer can be used to try to fill small gaps of 1-5 nt.

To do that, I should make a script to

1. paste exons with Ns,
2. run sealer and
3. split again into exons.

The module is completely silent

The threading is poor

Instead of providing once the bloom filter path, provide it twice, one for initial exon prediction, and a second one (if necessary) for sealer. This way, we can pipe it from gzip and save space, and maybe time.

Sometimes exons are contained inside other exons (ISI, A5 and A3 splicing events).

Maybe it is not necessary to do a pairwise comparison between all but inside a connected component

Currently, build_baited_bloom_filter does both the filter building and read classification (despite its name). Even though filter building is quite fast, it would be prudent to be able to re-use one filter for multiple read datasets. Perhaps add an option to only build the filter, and allow to specify a pre-existing filter?

In the README under "Running the pipeline":

2.:

• build_splicegraph should be build_splice_graph
• genome_k25_m100M_l1.bloom should be genome_k25_m500M_l1.bf

3.:
Neither gfa_to_exons nor gfa_to_gapped_transcript exist. Instead, gfa1_to_fasta is there and seems to have options that modify its behaviour according to the other two tools. Amend the instructions to reflect this.

We see that 2-4 GB are enough to process

read transcriptome when necessary, instead of storing the file the entire pipeline

collapse merges exons by exact identity.

A more successful aproach could be to map the exons to every exon, get blocks of mathces and recompose the graph.

should be faster, instead of yielding lines

• Make a small filter for the complete transcriptome
• split transcriptome into subtranscriptomes (~ 1000)
• run build_splice_graph in parallel, without collapse
• merge GFA1, collapse optionally

Matches between consecutive exons follow a Poisson distribution. In the merge2 step, we merge exons that overlap 8 or more consecutive exons. This happens, rarely, but happens.

match_distributions.pdf

Bloom filters are known for having false positives. In this context, the reported exons may have one or two additional bases at the start or the end of the exon. Therefore, we could give the option to the script to trim 1 or 2 nucleotides to each exon.