biothings / discovery-app Goto Github PK

For people who have already registered their dataset in a repository (like figshare, dataverse, NCBI GEO, etc.), would be nice to pre-populate our form with matching fields. Note that in some cases (e.g., figshare, dataverse) they will already expose structured metadata via schema.org standards. In other cases (e.g., NCBI GEO), we'd need to provide the mapping leveraging the work we did with metadataplus.

Suggested NIAID/DDWG 2020-10-20

I need to create a new class for which schema:Thing is probably the best starting point. Can't actually find schema:Thing when doing a search though.

When I try to edit a dataset's metadata, for instance to fix author.affiliation to make it an object, I get a validation error that creator is a required field. According to the schema, creator isn't required: author is. Maybe pulling the incorrect schema for validation?

error message:

'creator' is a required property
Hint: name,description,creator,publisher,identifier

Something like "create a new dataset record like this". Not sure whether to display that on the existing dataset details page, or have a dataset selector on the dataset registration page, or something else... This can be accomplished by downloading and importing the JSON, so this would be an easier way of accomplishing that...

https://discovery.biothings.io/view/niaid/NiaidDataset

Initial state of the toggles isn't synced. Clicking "show/hide all properties on page" doesn't work unless you uncheck the ⭐ "show all inherited properties"

It appears that of the 39 datasets currently at https://discovery.biothings.io/dataset, Google Dataset Search is only indexing the Wellderly dataset: https://datasetsearch.research.google.com/search?query=site%3Adiscovery.biothings.io

Not sure if it's the only reason, but rich results test on https://discovery.biothings.io/dataset/da4905854c18028d gives a "Page partially loaded" error...

To reproduce:

1. go to a guide form
2. "Import Metadata" --> "Metadata by you registered here" --> select a dataset
3. required field values should have pre-filled, but the "required field" error still there. These error msgs should be cleared.

e.g. for https://discovery.biothings.io/dataset/dcaab754073ad77b, display that it came from https://discovery.biothings.io/guide/niaid/

Based off of feedback from the 2020-10-20 NIAID DDWG working group demo, it seems like it'd be helpful to outline all the ways the /guide can be used. A few use cases:

1. Manually type everything into guide
2. Upload a .json for a single dataset (and possibly add/verify/augment fields)
3. Upload a .csv for a single dataset (and possibly add/verify/augment fields)
4. Upload from Github
5. Upload multiple datasets in a .json (and possibly add/verify/augment fields): see #12
6. Upload multiple datasets in a .csv (and possibly add/verify/augment fields): see #12
7. Modify a single property for a registered dataset (change in registered dataset list)
8. Modify a bunch of properties for a registered dataset (import in guide
9. Use an existing dataset as a template; for instance, load everything, then change identifier and make minor adjustments to other fields: see #34
10. Import dataset metadata from an NCBI repo and augment / conform to our schema: see #35

Using the DDE playground to create a new schema seems to work fine when 'extending' it from a schema.org class. The schema generated in the preview can be saved to github, loaded into the schema playground and correctly visualized. Example schema.

This is not the case when you try to extend from a (non-schema.org) schema registered in the DDE playground such as the outbreak schema or even the BioMedicalDataset schema. Example 1: bte:BioMedicalDataset schema-derived class. Example 2: outbreak:Dataset schema-derived class.

The new schemas created from the registered class will give the following error when you try to load them. "Failed because:
field about in $validation is not correctly documented"

on a dataset details page (e.g. https://discovery.biothings.io/dataset/da4905854c18028d) display who registered dataset. Even better to have a link to see all the other datasets that person registered...

for class page like this one: http://discovery.biothings.io/schema/Dataset

use case would be: to quickly check a known property using browser "find", which requires all properties displayed on the page.

Right now, the "Dataset registry" link is nested under "Discovery Guide". I think that should be its own top-level link (perhaps with guide-specific registry links nested underneath that, when that feature is available...)

Here is an example:

This is the original schema file:
https://discovery.biothings.io/api/registry/n3c/n3c:Dataset

This is the registered version accessed from the API:
https://discovery.biothings.io/api/registry/n3c/n3c:Dataset

extra prefixes like outbreak and niaid were added, which are not related to the n3c schema

When you search for a term against an ontology, it'd be nice to transfer the input search term to the "Didn't find what you are looking for? Enter value here:" input box.

Steps to reproduce:

1. https://discovery.biothings.io/guide/niaid
2. Add "metagenomics" as measurementTechnique
3. Idealized behavior: "metagenomics" would already appear in "Didn't find what you are looking for? Enter value here:" so you can change or submit.

After I enter all metadata using the NIAID guide, I get to this screen:

Clicking the "Register" link produces this modal:

Clicking "Register Now" seems to just make the modal disappear -- was expecting it to report back "Success" or "Failure" on a new page.

It appears that the dataset was not successfully registered since I do not see it at https://discovery.biothings.io/dataset

Suggestion from 2020-10-20 DDWG working group meeting.

When one enters a term that has a lot of hits, it's really hard to find what you're looking for in the long list of matching terms. For instance: if you enter "influenza a" for "infectiousAgent" in the NIAID dataset schema, the terms are cut off (since they're long):

Suggest:

1. Sorting the list by best match first; it looks like spaces are getting split in some cases (sorry, don't have an example), so you're getting partial matches for things like "influenza"
2. Add a search box at the top to filter the results. For instance, if you typed "Washington", it'd only give you things like "Influenza A virus (A/Washington2958/2012(H1N1))"

Right now, it's difficult to drag and drop from the validation options to properties further down the list.

Either enable scrolling of the list itself (so user can scroll to property) and then do the drag and drop, or allow the drag/drop menu to float so that when user scrolls down the screen, the drag/drop menu doesn't move out of view.

I got to this error state when I click the "Register" button:

This is what I filled out for required fields (left all recommended ones blank):

Currently, it looks like there is a "silent" error -- I would expect that any issue would be flagged in the web interface...

It's possible that users might try to register a dataset that already exists in the DDE, or they might want to work from a template of a similar dataset (see #34 ).

Similar to how Stack Overflow suggests you look at previously asked/answered questions rather than duplicating a question, it'd be cool as you enter data for it to suggest that the dataset might already be registered.

Suggested ways of detecting similarity:

• name / title (exact / Jacobian similarity)
• description
• author/creator

Error message: "field analysisCode in $validation is not correctly documented"

Steps to reproduce:
Try to visualize https://raw.githubusercontent.com/SuLab/niaid-data-portal/master/schema/Demo-Extended-Dataset.jsonld on https://discovery.biothings.io/schema-playground

Steps to create .json schema

1. https://discovery.biothings.io/registry
2. Search for a dataset (NIAID); select NIAID dataset
3. Create namespace, class
4. Add custom property analysisCode of type schema:SoftwareSourceCode

There are currently 39 datasets in the dataset registry (https://discovery.biothings.io/dataset). When I search "snohomish", I expect to get only one data set back (https://discovery.biothings.io/dataset/220edb57fb41eeea). The discovery app correctly highlights that dataset, but the API is currently returning all 39 (https://discovery.biothings.io/api/dataset?q=snohomish)

Steps to reproduce:

1. https://discovery.biothings.io/guide/niaid
2. try to lookup species or pathogen, using 9606, human, homo sapiens, etc.: https://www.ncbi.nlm.nih.gov/Taxonomy/Browser/wwwtax.cgi?id=9606

... --> no preview

The use case is some field in the schema is not intended for user-input.

If an array can be entered into the guide with a comma-separated list of values, provide helper text to user about this capability

but can be returned with meta=1/true query param.

The "branding" on https://discovery.biothings.io/portal/cvisb is a little muddled because we mix up CViSB and NIAID. Currently low priority, but at some point we should make that clearer...

1. In the Guide, sometimes the description of the property isn't listed for nested objects, e.g. for NIAID dataset:

2. Add asterisks or similar to make explicit which properties are required for objects within a schema (e.g. funder.name, identifier are required; rest are optional for NIAID:funding)

3. Specify the parent object in the field name somehow. For instance, NIAID has two descriptions: description and funding.description. When filling in the funding description, some users were confused about whether that was referring to the dataset description or the funding description

Based off of feedback from 2020-10-20 NIAID DDWG call.

e.g. for /guide/niaid, display https://discovery.biothings.io/dataset/21d8178fdf22998b, https://discovery.biothings.io/dataset/2aedff2a85b11474, ....

Bug: When registering a dataset, children seem to be inheriting their parent's marginality

Example: Why is url a required property on creator in NIAID? It should be optional according to the schema:
https://github.com/SuLab/niaid-data-portal/blob/master/schema/NIAIDDataset.json

Steps to reproduce:

1. https://discovery.biothings.io/guide/niaid
2. Add a creator; try to bypass url.

See also: funding:description, funding:url, funding:parentOrganization, ...

@flaneuse to generate the list of repos that schema.org compliant.

@marcodarko to figure out how to display on https://discovery.biothings.io/guide/niaid/

I have two suggestions to improve the dataset registration form:

• Form asks for "The identifier of the dataset if available", but it's a required field and the form won't allow moving on unless that's filled. So suggest removing "if available".
• It is unclear what "Reusable person definition" means.

🐛 when extending a schema that has re-used a schema.org property, if you make one of those properties required, it’ll duplicate it in the required object. This will throw an error when you try to visualize the schema. Need to de-duplicate the required list.

Steps to replicate:

1. https://discovery.biothings.io/registry
2. Search for a dataset (NIAID); select NIAID dataset
3. Enter namespace, class
4. Choose "name" to be re-used and required
5. Save .json, upload to GitHub
6. Try to visualized on https://discovery.biothings.io/schema-playground

When creating a new property, the schema editor provides an easy-to-use interface, UNLESS, you happen to make a mistake at the input type step. Since there's no way to remove an erroneous input type, you have no choice but to submit the erroneous property, delete it, and start all over.

The temptation is to click in the box listing the input types and to hit backspace (in order to delete the erroneous input type), which doesn't delete it, but can send you back one page (in the browser) which can potentially cause the user to need to restart.

This issue seems having two behaviors:

1. If accessing https://discovery.biothings.io/view/ctsa_blah/ for the first time, the page shows empty schema list, which is correct, but need a more explicit msg saying the namespace does not exist.

2. If accessing https://discovery.biothings.io/view/ctsa_blah/ after I viewed another schema (e.g. by clicking one of example links), somehow, the first link shows the content from the previous schema. Looks like it picked up the cached data from the local storage.

When there's an API event (someone added a schema, deleted a dataset, ...),

What we are interested in next, is to make this one way operation two way, and enabling things like:

1. record the response from the notification channel, and save it to the corresponding document's _meta field.
2. some notification channels may support interaction, for example, slack notifications can define buttons for users to click.
3. other long-live interaction sessions.

Note we are not planning on supporting anything but the first use case now, but ideally the structure should support further extension.

In the NiaidDataset, funding.funder, funding.identifier, and funding.funder.name should be required:

"monetaryGrant": {
"type": "object",
"@type": "MonetaryGrant",
"description": "Funding that supports (sponsors) the collection of this dataset through some kind of financial contribution",
"properties": {
"funder": {
"description": "An organization associated with a creator or funder of a dataset",
"oneOf": [
{
"$ref": "#/definitions/funder"
},
{
"type": "array",
"items": {
"$ref": "#/definitions/funder"
}
}
]
},
"description": {
"type": "string",
"description": "description about the funding award / grant"
},
"url": {
"type": "string",
"description": "award / grant URL"
},
"identifier": {
"type": "string",
"description": "Unique identifier(s) for the grant(s) used to fund the Dataset"
}
},
"required": [
"funder",
"identifier"
]
}

The display doesn't show this:

When registering a dataset (I think in the DataDistribution section), I see this dialog box:

To promote good data practices, suggest switching to YYYY-MM-DD to follow ISO 8601 standard.

To reproduce:

https://discovery.biothings.io/guide/n3c/dataset

author field:

• add two items for author
• reload the page
• only one item displayed
• the Add author button disappears

Use case: a user has a bunch of datasets to register in https://discovery.biothings.io/guide/niaid/. Rather than submitting them one by one, it'd be cool to allow users to be able to register multiple simultaneously.

Ideally the inputs would be a .json array or a .csv/.tsv file.

Right now, I believe the form defaults to Person. I think it would be better if users have a choice between Person and Organization.

Would be nice in https://discovery.biothings.io/view/niaid/NiaidDataset (or other schemas) to have a link to download the schema as .jsonld file.

https://discovery.biothings.io/editor shows both variableMeasured (right) and variablesMeasured (wrong) when you try to extend schema:Dataset.

CreativeWork shows 89 inherited properties, but I count 103 on the current Dataset page (counting by hand so might be off by +/- 1).

Suggest pulling schema.org properties from their latest definition at https://schema.org/version/latest/schemaorg-current-http.jsonld (schema.org for developers)

It seems to give the array of strings instead.

In NIAID Dataset, the infectiousAgent property is shown as a CreativeWork with type string and Non-Strict term lookup. It should just be a (non-strict) enumerated string, not a CreativeWork object.

Currently, the "Discovery Guide" link in the header goes to https://discovery.biothings.io/best-practices. A few possible suggestions:

1. Is /best-practices the best location for this page? "Best practices" and "Discovery Guide" seem like two different things...

2. At the top of that page, perhaps have a short text description of what a "Discovery Guide" is (not obvious from that phrase alone). Basically explain that it is a wizard that allows dataset registration according to a given schema.

3. Now that we have three guides, have links on this page to all three of them

A login link like this:

https://discovery.biothings.io/login?next=/guide/n3c/dataset

If go with the github login, the redirection follows the url passed to next parameter, however, the login via UNA does not.