hubmapconsortium / ingest-validation-tools Goto Github PK

We need to generate user facing documentation from the schema.

• Nils suggests OpenAPI, but that is for REST APIs, not simply schemas. And the presentation it has for schemas is not much better than looking at source code: petstore demo
• CIDC does use JSON Schema for their manifests, and I don't see anything about doc generation there.
• The JSON schema site itself has code generation, and form generation, but no doc generation.

We want to make sure the the data_paths are not only unique, but that none are left substrings of others. (ie, the trees don't overlap.)

For ATACseq data, metadata columns cell_barcode_offset (column AE) and cell_barcode_size (column AF) take 3 comma-separated values.
Cell barcodes are 3 x 8 bp (for UCSD, it's 3 x 8bp) sequences that are spaced by constant sequences (the offsets).
Barcode size: 8,8,8
Barcode offset: 0,38,76
First barcode at position 0, then 38, then 76.

Delete the metadata fieldname:

"subspecimen_assay_input_number"

on line 124 of the yaml. This information is captured by another filename.

In the first metadata.tsv we got (from Dinh), they’re expecting to be able to use globs (actually, [1-3] , but *-globs could work) for datasets, instead of having neatly separated subdirectories. Would this be feasible to handle downstream? I’d really prefer not, but if it would be feasible for you, I don’t want the validation to be an arbitrary stumbling block.

Will you be able to pass in a local filesystem path for the submission, or will it be something else… maybe I need to make globus API calls, or maybe you’ll pass me a directory listing, or maybe …?

... in particular: The protocols.io field got a different name in the atac spec.

Test passed locally, failed on travis... guessing it pulled from some OS localization file which wasn't present there?

I can think of two main alternatives, though there may be others:

• Keep this repo: publish to PyPI, and the ingest pipeline will use it.
• This whole repo is assimilated into the ingest pipeline.

The allowed values are now:

• 10X snATAC
• In-house produced (Protocol Reference)
• Nextera

change

• In-house produced (Protocol Reference)
  to
• in-house

cell_barcode_offset definition should read:
Positions in the read at which the cell barcodes start. Cell barcodes are 3 x 8 bp sequences that are spaced by constant sequences (the offsets). First barcode at position 0, then 38, then 76. (Does not apply to SNARE-seq and BulkATAC)

cell_barcode_size definition should read:
Length of the cell barcode in base pairs. Cell barcodes are, for example, 3 x 8 bp sequences that are spaced by constant sequences, the offsets. (Does not apply to SNARE-seq and BulkATAC)

For codex submissions, I’m currently distinguishing akoya and stanford… it could be merged, but that would make it a lot harder to understand the error messages.

I think this might be the location?

https://docs.google.com/document/d/1UqAq04xSzd7PhhdXaiMVUk-lVaZB1orGXvyFS5CHKI8/edit

Matt Ruffalo 1:19 PM
UCSD dataset ea42a70873cd0eb7e2cc91fd6d79fc8b

The same fixture might be used in multiple doctests, but I know I have gotten confused when I've duplicated a file to make a new doctest, and then forgot to change the target inside.

The issue regarding transposition_transposase_source has been resolved but the TODO note is still there causing confusion to viewers.

@cebriggs7135 : Looping you in: I think it's not feasible to support timezone abbreviations in the datetimes. The three letter codes are just not specific, maintaining our own list of HuBMAP approved timezones will be an on-going nightmare, and the code needed to parse the date becomes much more complicated at every level.

As an alternative, I think folks can just use the +/- notation: +06:00 for example.

https://docs.google.com/document/d/1Lrzruebio9nUusaFSJQlZGANkRjS5lrJmk4W0sSj_xM/edit#heading=h.tyjcwt

Look at globus and the codex submission spec.

The original pointer was to cidc-schemas, but I have some concerns after looking at an an example:

• It's not generating a template from json schema: This is their own syntax, though it looks like it does reference types in json schema.
• The pypi package is pretty huge, with lot of dependencies, and when I start up there are warnings about deprecations.
• The template that it produces is Excel, not CSV.

Alternatives to:

• Write our own JSON Schema.
• Table Schema and its python implementation: This looks well thought out, and in a JSON syntax supports things like unique values across columns, and referential integrity. Used by outside projects like csvlint.
• goodtables-py is under the same umbrella: Not sure how it relates.
• PandasSchema would require pandas, and the schema is python.
• CSV Schema from LoC only seems to have Java implementation: csv-validator

For completeness:

• pandas-validator is archived.
• CsvSchema hasn't been touched in six years.
• CsvValidator hasn't been touched in seven.

I think a key-value table would work well.

The generated documentation would be more useful if it could be part of a larger document: Consider adding pre.md and post.md which could be used to bracket the generated content in a larger document.

Just an explanation of the generate and validate scripts: What their input and output is.

This field has a TODO question:
TODO: Should this be updated? Seems like copy and paste from CODEX.

The answer is no, it should not be updated. CODEX is just offered as an example of a targeted assay.

Make sure the donor and sample IDs passed in match what we see in the file.

We want to be sure that all the files are referenced by exactly one data_path.

The CODEX metadata ingest form has 2 fields for data_path and metadata_path. Please add the same 2 fields for submitters to paste the Globus link to their data and a link to metadata.

• antibody id
• antibody name
• protein name (https://www.ncbi.nlm.nih.gov/genome/doc/internatprot_nomenguide/)
• cycle number
• channel number
• link to metadata tsv (use the path as the ID)

Right now, I have validate.py which takes command line arguments, and either exits with 0 on success, or non-0, and with human-readable errors on STDOUT… is that the easiest interface for you to use, or would something else be better?

We want to make sure the data_paths actually exist in the submission

Table schema is richer, but we can get some of the constraints encoded in the json schema as well.

• Number the errors
• Give an explanation for "oneOf" and "contains"

They are just to keep ugly <BLANKLINE> out of doctests.

This (or documentation generated from this source code) needs to be the one place people look for how to structure their submission. Higher level concerns can still be described in docs, but the details need to be here, and only here.

• First, I need to talk with Joel and figure out whether this will continue to exist independently, or if it will be merged into another repo.
• With that settled, other docs can be simplified so that they point here for details.
• Example spreadsheets should preferably be removed entirely, or if necessary, retitled to indicate they are deprecated.

We know the standards are going to continue to evolve, and this is the way I think it can work. If you see a problem with this course, please raise an alarm!

The 3 options should be:

1. microscopy
2. mass spec
3. sequence

Should I put this on PyPI, or would you like something else?

Telling someone the error is in column 46 on a spreadsheet isn't helpful.