Import the abundance tables via http://biom-format.org/ rather than plain text.

There's nothing to stop us streaming the zip download, rather than generating the whole (quite large) thing in memory. I've found this Python module which will do the job.

https://github.com/allanlei/python-zipstream

Use NCBI BLAST to search a fasta file of the database.

We're not mapping the ontologies through into string form.

... with a link to the password reset page. Jeff has a card in the RDC project.

Urgent, but simple fix.

If you subset the data (e.g. to just environment = Soil), many of the contextual metadata fields become irrelevant and will be blank for all samples.

We already deal with this in the biom export, by doing a first pass to determine the interesting fields, and then writing only those fields into the BIOM file. We need to do the same thing for the CSV export code.

Take a look at the code and have a go at making the change. If you look at biom.py you'll see how we approach the issue for the other output format.

Make the filenames for the Zip export of OTU data (CSV and BIOM) meaningful.

• improve the outer filename (the .zip)
• improve the inner filename (only applicable to BIOM)

Perhaps we should just include the date and time in ISO format, with a sensible prefix. So, 'BiomExport-20180524T1406.zip'. Note we drop the colons from the ISO format as they'll cause problems on macOS.

Reported by Jeff:
Modify BIOM format produced from CCG so that the value of "format" = "Biological Observation Matrix 1.0.0" (and not "1.0.0" as it is now)

Steps to reproduce:

• add a contextual filter with a combo box (e.g. Australian Soil Classification)
• leave as defaults
• press the search button
• error will be shown

I've investigated, the underlying issue appears to be that when we emit selectContextualFilter we add a clone of EmptyContextualFilter to the redux store. This doesn't actually match what we're displaying in the UI - in the UI we're showing the first combo box option as being selected.

I'm not sure the best approach to fix this. We'll probably need a different template EmptyContextualFilter for each type of filter (string, combo box / ontology, float range, ...)

@sztamas when you're back could you maybe suggest a fix?

Phyloseq Installation
https://joey711.github.io/phyloseq/install.html
source('http://bioconductor.org/biocLite.R')
biocLite('phyloseq')

The phyloseq uses import_biom to import BIOM file.
e.g.,
x<-import_biom("GP.biom")

An issue occur when using the same feature to import the following biom file.

x<-import_biom("json_biom_file.biom")

Error in dimnames(x) <- dn :
length of 'dimnames' [2] not equal to array extent
In addition: There were 50 or more warnings (use warnings() to see the first 50)

Warning messages:
1: In parseFunction(i$metadata$taxonomy) : Empty taxonomy vector encountered.
2: In parseFunction(i$metadata$taxonomy) : Empty taxonomy vector encountered.
3: In parseFunction(i$metadata$taxonomy) : Empty taxonomy vector encountered.
4: In parseFunction(i$metadata$taxonomy) : Empty taxonomy vector encountered.
....

Some phyloseq users have posted the same issue here.
joey711/phyloseq#674

The contextual metadata is public, so there's no need for auth on this page.

Looks like it's related to commit 70a78ad.

The egg-link from /env to bpaotu in /app references non-existent path /data/app.

This is affecting staging.

A lot of the bacteria data doesn't seem to be making it into the database. The methodology is unchanged, we just have more input data into Andrew's pipeline, so we should have more data, or the same amount, but nothing that was there before should be gone now.

Example missing data point:

Sample ID: 7031
OTU: AACAAACGCTGGCGGCGTGCCTAATACATGCAAGTCGAACGAGGAAGTCCCTTCGGGGATTGTACTAGTGGCGTACGGGTGAGTAACGCGTGGATAATCTTCCTTAAGGTGGGGAATAACTAGTCGAAAGATTAGCTAATACCGCATAAGACCACAGGCTCTTCGGAGCAAGGGGTTAAAGCCGAAAGGCGCCATAAGATGAGTCTGCGCCCGATTAGCTAGTTGGTGAGGTAATGGCTCACCAAGGCGACGATCGGTAGCTGGTCTGAGAGGACGGCCAGCCACACTGGGACTGAGACACGGCCCAGACTCCTACGGGAGGCAGCAGTAGGGAATATTGCACAATGGAGGAAACTCTGATGCAGCGACGCCGCGTGAGTGACGAAGGCCTTCGGGTTGTAAAGCTCTGTTCTCAGGGAAAAAGAAAGTGATTGTACCTGAGAAGAAAGGACCGGCTAACTTCGTGC
Count: 3

webapp=# select otu.code, sample_otu.count from sample_otu join otu on otu.id=sample_otu.otu_id join ontology_otuamplicon a on a.id=otu.amplicon_id where sample_id=7031 and otu.amplicon_id=3 and otu.code='AACAAACGCTGGCGGCGTGCCTAATACATGCAAGTCGAACGAGGAAGTCCCTTCGGGGATTGTACTAGTGGCGTACGGGTGAGTAACGCGTGGATAATCTTCCTTAAGGTGGGGAATAACTAGTCGAAAGATTAGCTAATACCGCATAAGACCACAGGCTCTTCGGAGCAAGGGGTTAAAGCCGAAAGGCGCCATAAGATGAGTCTGCGCCCGATTAGCTAGTTGGTGAGGTAATGGCTCACCAAGGCGACGATCGGTAGCTGGTCTGAGAGGACGGCCAGCCACACTGGGACTGAGACACGGCCCAGACTCCTACGGGAGGCAGCAGTAGGGAATATTGCACAATGGAGGAAACTCTGATGCAGCGACGCCGCGTGAGTGACGAAGGCCTTCGGGTTGTAAAGCTCTGTTCTCAGGGAAAAAGAAAGTGATTGTACCTGAGAAGAAAGGACCGGCTAACTTCGTGC';
 code | count 
------+-------
(0 rows)

this vs the old dataset:

bpaotu=# select otu.code, sample_otu.count from sample_otu join otu on otu.id=sample_otu.otu_id join ontology_otuamplicon a on a.id=otu.amplicon_id where sample_id=7031 and otu.amplicon_id=3 and otu.code='AACAAACGCTGGCGGCGTGCCTAATACATGCAAGTCGAACGAGGAAGTCCCTTCGGGGATTGTACTAGTGGCGTACGGGTGAGTAACGCGTGGATAATCTTCCTTAAGGTGGGGAATAACTAGTCGAAAGATTAGCTAATACCGCATAAGACCACAGGCTCTTCGGAGCAAGGGGTTAAAGCCGAAAGGCGCCATAAGATGAGTCTGCGCCCGATTAGCTAGTTGGTGAGGTAATGGCTCACCAAGGCGACGATCGGTAGCTGGTCTGAGAGGACGGCCAGCCACACTGGGACTGAGACACGGCCCAGACTCCTACGGGAGGCAGCAGTAGGGAATATTGCACAATGGAGGAAACTCTGATGCAGCGACGCCGCGTGAGTGACGAAGGCCTTCGGGTTGTAAAGCTCTGTTCTCAGGGAAAAAGAAAGTGATTGTACCTGAGAAGAAAGGACCGGCTAACTTCGTGC';
                                                                                                                                                                                                                   
                     code                                                                                                                                                                                          
                                               | count 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------+-------
 AACAAACGCTGGCGGCGTGCCTAATACATGCAAGTCGAACGAGGAAGTCCCTTCGGGGATTGTACTAGTGGCGTACGGGTGAGTAACGCGTGGATAATCTTCCTTAAGGTGGGGAATAACTAGTCGAAAGATTAGCTAATACCGCATAAGACCACAGGCTCTTCGGAGCAAGGGGTTAAAGCCGAAAGGCGCCATAAGATGAGTCTGCGC
CCGATTAGCTAGTTGGTGAGGTAATGGCTCACCAAGGCGACGATCGGTAGCTGGTCTGAGAGGACGGCCAGCCACACTGGGACTGAGACACGGCCCAGACTCCTACGGGAGGCAGCAGTAGGGAATATTGCACAATGGAGGAAACTCTGATGCAGCGACGCCGCGTGAGTGACGAAGGCCTTCGGGTTGTAAAGCTCTGTTCTCAGGGAAA
AAGAAAGTGATTGTACCTGAGAAGAAAGGACCGGCTAACTTCGTGC |     3

Dockerised yarn, webpack and es6 or typescript for a happier life.

Drop the '_id' from the title.

When selecting the Soil Environment in the Contextual Filter section, contextual filters of type Soil aren't shown in the Contextual Filter list boxes.

Reported by Anna Fitzgerald:

If you query the database for "Broad Land Use" - "Conservation and Natural Environments - National Park" and then hit search, no results are found.

That's suspicious - we need to look into why there are no matches.

Ideally, the hierarchy in this ontology should be respected when querying. So searching for "Conservation and Natural Environments" should return all samples which have that key set, or any of the child keys. This may mean that we need to change the ontology columns from integer type to a set type, so we can do "IN" queries on the column.

Steps to reproduce:

• run with CKAN integration turned on, on the same host and port (e.g. localhost:8000)
• disable CKAN integration (or swap from dockercompose-bpa-ckan to plain bpaotu)
• browser will still be sending the cookie through; this seems to cause 403s

Workaround is to use private browsing mode; then everything works fine.

This is a low priority as it only affects developers.

See linked trello card for details.

• sort the metadata fields
• always the same fields in each entry, even if the value is 'null'

Unsorted column header of metadata table

The first thing that I have noticed is the key order (column header of the metadata in this case) that is not alphabetical sorted when extracting using json.load in python. The result produced by python2 and python3 is not the same.
Having an alphabetical order of column header is important for feeding phyloseq wrapper with correct column number when creating NMDS plot.
Regarding to this finding, I have posted an issue on biom-format.org github after going over the python script (parse.py) on biom-format github.

Missing values in metadata table

The next thing I found is the metadata table containing missing values which I think it’s reasonable, but it will produce inconsistent key/value pairs of each sample inside JSON structure.
With different key/value pairs in JSON structure, it might be the issue confusing the phyloseq with missing metadata table.

Put a clear "return to the Australian Microbiome home page" link somewhere in the UI.

Link to https://data.bioplatforms.com/organization/about/australian-microbiome

• only a few hundred samples have contextual metadata
• this causes most of the abundance data to be excluded from the ingest

The Submit to Galaxy functionality is failing currently if the galaxy user doesn't exist.

The view is calling galaxy_client.create_user on the galaxy client, but the client has been changed so the new function is galaxy_client.user.create.

This happened when we moved to React, it should be simple to add this feature back in. I'll take a look, it'd be good to get across the new React stuff anyway!

Reported by Jeff from QCIF.

Port over to the new react frontend stack.

There's a line of samples off the east coast of NZ. They're not showing up in the default zoom, but if you zoom out enough so that the world map wraps, you'll see them appear. I wonder if it's some issue with them being too close to the wrapping point in longitude?

Jeff provided a new version of the Galaxy Quick Start Guide.

Update to it.

When samples are returned from a query of the BPA Data Portal, in the summary list, also include a column for each of the contextual filters included in the search

This makes it easier for the user to review the results and satisfy themselves that the results are correct.

e.g. if one stipulates a search of vegetation type = grassland and vegetation type = forest, a column called "vegetation type" should appear and only include the values of grassland and forest.

We're writing out the ontology numbers (which are meaningless) into the BIOM file. We should instead write out the textual version, by doing an onotology lookup.

Export data in the BIOM format (hdf5)

http://biom-format.org/

We can't stream BIOM data, so we'll need to prepare it on the server and then email the person a link.

Start off by reviewing the BIOM format docs, and then come back with a proposed way to add this support. We must keep the current CSV format support, this is an additional feature.

Note this is required for the RDC project to move forward.

Set up webpack and django to have hot-reloading while developing.

Also, add a hash to the js bundles generated by webpack, so we invalidate the cache when publishing a new version of the app.

See linked trello card.

The function doing the verification returns HttpResposeForbidden but the code invoking it is assuming an exception is thrown.

e.g. the contextual, amplicon and taxonomic filters which have been applied.

this will then flow through to end users via tooling (e.g. galaxy/phyloseq)

See trello card: https://trello.com/c/sKn1cp7L/141-modify-biom-format-from-ccg-so-that-it-is-the-same-as-a-phinch-format-file-which-can-be-read-into-both-phinch-and-krona

https://bpa-staging.ccgapps.com.au/bpa/otu/

Hi Grahame,

A bug has crept in – when a dataset has been sent to Galaxy, the blue blox (Submission to Galaxy in Progress…” no longer changes to the green box with the message that includes the link to the Galaxy History.

Can you please fix?

Add broad land use, get this traceback

runserver_1 | sqlalchemy.exc.InvalidRequestError: Could not find a FROM clause to join from. Tried joining to <class 'bpaotu.otu.SampleLandUse'>, but got: Can't determine join between 'Join object on sample_context(140241661005216) and ontology_sampleaustraliansoilclassification(140241661226288)' and 'ontology_samplelanduse'; tables have more than one foreign key constraint relationship between them. Please specify the 'onclause' of this join explicitly.

• show sample contextual metadata when a point is clicked upon in the map
• if there are two points at the same (lat, lng) but different depths, show them in as tabs, and only show a single point

The kingdom filters always work, but the second-level is either too slow, or not working at all in some cases. We need to track this down and resolve.

We are receiving bug reports from users on this, and it happened to me during a demo, so it's becoming urgent.

• make the map reachable via a direct link, e.g. /map (relative to the base URL for the app)

Just show all the samples.

The BIOM export is working, but is a little slow.

Investigate performance improvements.

Most of the data output should be coming from the abundance_tbl function; we should heavily optimise this function. I suspect the call to format() at line 96 of biom.py is actually more expensive than we'd like. We could change the calls to .index() to use a O(1) lookup list rather than the current O(N) approach.

Once we've uploaded the exported BIOM data to Galaxy, provide a link to that history to the user. Probably use the existing "it worked" status message, but remove the timeout and add the link over to Galaxy.

• add text box (sequence to search) and button for BLAST under taxonomy box.
• enable text box only if Amplicon is provided in the search form.