multiqc / multiqc Goto Github PK

Now that the original images are going, there aren't actually that many assets required for the reports - just lots of CSS and Javascript.

It would be nice if we could embed all of this into the HTML file, so that it's easier to share. Preferably using a Python package to do this at run time to avoid maintenance issues.

The 'Input filename' string that seems to be used as an indicator of cutadapt logfile (i.e. https://github.com/ewels/MultiQC/blob/master/multiqc/cutadapt/__init__.py#L66) is not present in my cutadapt log... I used cutadapt 1.8. It gives the line "Command line parameters: ..." instead.

Is it because I run it in paired end mode, i.e. use two input files?

Cheers,
Johannes

Would be cool if you could click table headers and get the table to sort by that column.

Instead of printing parsed data to the HTML as currently done, print the parsed data to a single JSON file per module, and then load this with ajax. Keeps the HTML cleaner and makes it easier for others to use the data.

Maybe also copy out the raw data file where applicable? eg. FastQC.

If reports are generated with mega-numbers of samples, the page can take for ever to render and even hang.

It would be nice if the javascript plots could check the number of samples. If it's huge, display a button instead. Clicking the button generates the plot.

This should massively improve page load time but still retain all functionality.

It seems that the hide samples regex on / off state isn't saved properly.

Sample names aren't always intuitive - it would be nice to have a tool to swap them out for something more useful. Needs to be easy to use with copy and paste.

Merged three issues into this:

1. Currently, if samples are hidden in the saved config, the page loads then is immediately unresponsive whilst those samples are removed.
2. Highlight line graphs samples - save doesn't work (#54)
3. Speed up 'hide samples' (#53)

Most of the toolbox features currently work on rendered plots - looking through existing features and changing them. Instead, it might be better to replot everything when something changes.

• Restructure plotting functions to work off a canonical data source that doesn't change
  • Make a copy of this data at plot time
  • Get highlight / rename / hide settings from page
  • Recolour / rename / remove data from the copied data structure
  • Create plot
• Refactor highlight / rename / hiding to call plotting functions when changed, instead of modifying directly
• Plotting functions should not fire on page load, instead be triggered by config load.

This is sort of semi-implemented already in some places, for instance all of the code involving sample renaming. This is a fairly large refactoring, but I think that it will make the reports quite a bit more stable.

It could also lead to a potential fix for the mysterious slowness in #53, as hiding samples often seems slower than opening the page in the first time.

Finally - it should also fix #54 as the sample colouration will be set properly in the highcharts settings, so properly exported in other file formats (rather than a post-plotting hack).

Get #10 to work, then write such an extension for the NGI (partly as an example for others).

Could include:

• A new theme (softlinking files, new javascript file to power API to rename samples).
• New super-specific module that checks the database for some kind of value?
• Could also add a new click argument? To e-mail to a user or something..?

Qualimap is widely used for data QC and would be a good addition to MultiQC. It has a number of data outputs that could be used, but I think it would be good to start with those plotted in ngi_visualizations...

General stats table:

• Median coverage
• Median insert size
• % of genome >= 30x coverage
• mapped reads avg. GC

Main sections:

• Coverage Histogram
• Insert Size Histogram
• Genome Fraction Coverage
• GC distribution

Medium - large reports now have significant file sizes, due to saving all of the original FastQC plot images. Whilst not consuming vast resources yet, it's getting to the size where they can be annoying to send by e-mail.

Removing the original images means that the reports should be back to being only a few hundred KB. Not sure that their use justifies the problems that they cause? Feel free to comment below if you disagree. Will leave this open for discussion for a bit.

The tools are currently a bit hidden off the bottom of the navigation, and as more modules are added / the tools get longer, they're going to get more and more obscured.

Maybe move them into a side-pull space on the right of the page?

To avoid overwriting sample names, give the option to prefix the directory name to every sample.

See the tutorial. Maybe wait a bit for it to settle down and become more stable first..

It would be nice to be able to sort samples in the general stats table by highlight. Would help #28.

Add a button above the table to do this once highlights have been applied?

It would be cool to have a sidebar text box which you can type into with the option to either show only or hide any samples which match that text..

A barplot showing the proportion of bp/reads passing each step of a Methyl NGS pipeline from:

1. Initial fastq.gz PF reads/bases
2. Qscore and adaptor filtering
3. Mapping
4. Deduplication
5. Clip Overlap
6. 5' or 3' trimming before methylation calling
7. Bases in CpG positions covered below 1x, above 1x, above 5x, above 30x

Hi again,

I seem to be naughty and rename the output .zip archives from fastqc (its done by our workflow to avoid outputfiles be named P1994_101_R1.fq_fastqc.html, but instead P1994_101_R1_fastqc.html when running on .fq.gz files), this causes confusion in multiqc since the dirname within the zip archive is still the old one...

When multiqc tries to look for the fastqc_data.txt file within the .zip it results in a KeyError since the directory name within the zip is wrong. I guess one could argue this is malpractice or a problem with fastqc, but it would be nice if multiqc could be tolerant of renaming the zip archives. I think I can sketch on a pull request for this if you may assist with comments and you think its a good idea.

Thanks,
Johannes

Hi Phil,
just started using your tool, great stuff. I was wandering if it is possible to fix the header in the General Statistics section in a way that follows you while scrolling down the page, since it easy to get lost in the middle of the sample list and forget what is what.
Thanks a lot and all the best!
Cheers

Edo

Have the option to share a single colour scale. For instance, all columns with read counts (data-scalename='readcount'). Needs to work across modules and be compatible with one or many columns. Probably take the first colour scheme specified.

Advantage - makes it clearer where reads are being lost through a selection process (for example).

We have Pass / Warn / Fail statuses from FastQC for every sample in every plot. Currently, it's tricky working out which lines have which status though. Could be nice to have a button that colour codes the data series by their status.

The option to show the original plots is very useful, but it breaks if you move the MultiQC_report resulting directory to a location where you don't have access to the original plots. i.e I ran MultiQC in our cluster, downloaded the resulting zip file to my laptop for better visualization, and the FastQC plots (the original ones) are not there.

What about something like --copy-original?

It would be ace to have a sidebar widget where you could specify the colour of the label for all samples matching a text input.

This would enable easy bulk labelling of certain groups - eg. all trimmed (*_trimmed) samples.

I like the FastQ Screen plot, as it mimics that made by the original tool. However, it gets a bit useless if there are lots of samples or species. So, in these cases, use a different plot..

Probably sum the various types of alignment score, then have a single bar per dataset, coloured by the species it aligns to.

I would say that the limit of usefulness is around 20 samples for 8 species. So, maybe count num_samples * num_species and if that is > 160 use the alternative plot style.

Someone mentioned what the best way would be to share 3rd party modules at DNA club

Maybe people are getting sick of me recommending setuptools entry points but I think they could prove a good fit in this case

Basically it would allow other people to build modules as Python packages that can be installed using pip. Whatever packages are active in the current env are the once used by MultiQC 😃

It was a nice idea, but it annoys me quite frequently and I've never found it useful to date.

Instead, have two buttons to ask for a save - one for this report only, one for all reports.

The Picard modules I would be interested in are

• CollectInsertSizeMetrics
• CollectGcBiasMetrics
• CollectAlignmentSummaryMetrics
• CalculateHsMetrics (see example from Pekka)
• CollectRnaSeqMetrics
• CollectOxoGMetrics
• BaseDistributionByCycleMetrics
• CollectWgsMetrics
• RrbsSummaryMetrics
• QualityByCycleMetrics
• QualityDistributionMetrics

Any other (picard) modules that are interesting to have?

Currently, when hiding samples it's possible to get a graph with no data. Instead of this, slot in an explanation saying that all samples have been hidden.

Would be nice to have a button which launches a modal showing the colour schemes used in the general stats table.

Hi, just a humble suggestion can you start a new project named MetaMultiQC (suggested by @guillermo-carrasco) (or possibly MultiMultiQC).

Just kidding. Do you think its a good idea for support for linking samples together?

I'll try to explain what I mean:
My usecase is that I run fastqc for each sample at each step I take in the processing of the reads. Below you can see the results for three samples in two steps = 18 fastqc reports.

Would be nice to see a graph over how the stats change for a sample between the different steps for example. In order to do that I need a way to link the reports that belongs to the same sample together.

Thanks,
Johannes

Highlight all samples in light grey first if any highlights are created. Disallow blank highlight search strings.

Could create another setuptools entrypoints class (multiqc.hooks.v1?) that could create a list of functions to run at different points in the code.

For example, for a paired-end bam file, preseq lc_extrap can be run like this:

Input file(s):  SAMPLE.bam
Output file(s):   SAMPLE.preseq.lc_extrap.tsv
Tool(s):        preseq lc_extrap

$ preseq lc_extrap
    -B –P  
    -o SAMPLE.preseq.lc_extrap.tsv  
    SAMPLE.bam

The output of preseq lc_extrap can be plotted in a curve indicating the ratio of “number of molecules (M)” and “number of distinct molecules (M)”. See below a description of three examples from the preseq website (http://smithlabresearch.org/software/preseq/challenge/)

Instructions on how to install this software and its dependencies can be found at http://smithlabresearch.org/software/preseq

It would be good if the toolbox buttons that have active options are highlighted in some way - a coloured line at the bottom of their button perhaps?

Would be even better still if this only appears when their filters are actually doing something. eg. if there's a renaming filter that has no effect, don't highlight. This is probably a bit tricky and unnecessary though.

With lots of samples, this can make the browser hang. Try to figure out what's taking it so long.

Make MultiQC support the plots that it currently ignores..

• Per base sequence quality
• Per tile sequence quality
• Per sequence quality scores
• Per base sequence content
• Per sequence GC content
• Per base N content
• Sequence Length Distribution
• Sequence Duplication Levels (needs thought)
• Overrepresented sequences
• Adapter Content
• Kmer Content (needs thought)

The toolbox doesn't float above the charts, some z-index thing:

• Browser: Chrome Canary
• Modules: General and FastQC

There is currently no regex mode option for renaming samples. Seems a bit unfair.

Something like this:

# update the context with new defaults from the config file
context.default_map = context.obj

Currently, each module has to write its own HTML for the general statistics table, which is messy.

Instead, write a function in the BaseModule to handle this.

Modules ported:

• bismark
• bowtie1
• bowtie2
• cutadapt
• (fastq_screen)
• fastqc
• featureCounts
• picard
• qualimap
• star
• tophat

At the moment, every template is stand alone. This means that if you just want to change the logo, or a bit of text in the report, then you have to copy everything from general and then make your change. Inevitably, this leads to drift away from the core package and theme.

Instead, allow templates to behave as 'child' themes. Could set a variable parent in the theme __init__.py with the name of another template and build off that. Then the parent template would be used, with any files in the child copied over the top of this. This could happen at run time, also allowing plugin packages to keep their code base clean.

To make this work, I need to split up the template into many more files instead of a single massive file. This is thankfully very easy with Jinja2 - see the docs.

Also, whilst the new stand-alone reports are very nice, we probably need to allow themes to create non stand-alone reports if they want to. Again, an option in the report's __init__.py file here I think.

Task list:

• Split general Jinja template into multiple files, centred on a base.html with blocks
• New optional variable in template __init__.py to specify parent theme
• New method of report generation, involving copying template files to a temporary location
  • See tempfile.mkdtemp()
• New template __init__.py config option to prepend a directory name to generated report
• New template __init__.py ability to copy files along with report generation.

Have a collapsible plot (hidden by default) below the general stats table. Use a select drop-down box to choose a column from the table, then plot that data. Respect ordering of rows. Use data-max and data-min to set plotting limits (unless data exceeds them). Think about plot types - presumably use a bar plot.

Would be a nice to have a "clear all" option for filters, especially when loading a page that has been played with previously. Maybe on the save page?

parsing output/stats from bwameth:
https://github.com/brentp/bwa-meth

Trimmomatic, yes. Also one of those tools that writes to stdout or a user-given log file.

Great initiative Phil!

I'm not going to work on this for now, but it would be interesting to integrate this byzantine mashup of R & Perl scripts in your mix:

http://labs.oicr.on.ca/Boutros-lab/software/SeqControl/

It generates fairly pretty QC plots and stuff ;)

Regex switch is currently just text. Make it look more obviously like a switch.

Allow multiple analysis directories to be given (positional, not flag arguments - see this)

When exporting line plots which have been highlighted, the line colours aren't respected. Maybe if we do this differently and replot it might work.

Doing non-standard stuff to highlight the bar plot, so don't think we'll get that to work any time soon.

Have a think about how to get the page setup to persist. Could be browser-generated javascript file download? Instruct the user to save it in the report directory. Then always try to load this file in case it's there..

Also could use cookies, though doesn't transfer across users. Might be cleaner. Could do both..