microbiomedata / dataharmonizer Goto Github PK

I moved test_sntc.py into tests/, and when I next tried to run the test suite (by right clicking on the rest folder in PyCharm), it appeared that a new google sheet authentication step was required

Please go to this URL and finish the authentication flow

etc., but the test script didn't pause for the next step:

Enter the authorization code:

I ran this in the PyCharm Python Shell, after cd-ing to tests\

import pygsheets
sntc_id = '1pSmxX6XGOxmoA7S7rKyj5OaEl3PmAl4jAOlROuNHrU0'
client_secret_json = "../local/client_secret.apps.googleusercontent.com.json"
gc = pygsheets.authorize(client_secret=some_google_auth_file)

Please go to this URL and finish the authentication flow: https://accounts.google.com/o/oauth2/auth?response_type=code&client_id...
Enter the authorization code: >? ...

That created a sheets.googleapis.com-python.json in tests/ and the tests continued successfully. Maybe I could have just copied the previously created sheets.googleapis.com-python.json from the project root?

Creating a sheets.googleapis.com-python.json requires a client secrets file, as defined in client_secret_json = "../local/client_secret.apps.googleusercontent.com.json" above. Neither of those should be checked into GitHub!

That conversion is performed by by script/make_data.py

Error: parent class history/tillage doesn't exist as section or field for term. Make sure parent term is trimmed of whitespace. zonal tillage

• from SNTC to LinkML
  • slots within the soil_biosample class in the annotated soil biosample YAML file
  • slot/column sections as LinkML annotations
  • slot/column order as LinkML annotations
  • section definitions and section ordering in LInkML ???
• from LinkML to DH data.tsv

Implement within linkml_to_dh_light.py for the nmdc_biosample_slots and mixs_packages_x_slots tabs

implement within inject_supplementary for all other tabs

• min
• max
• multivalued
• is_identifier (forces unique)

For example, now that @subdavis committed #15

Make a backup?

script/make_data.py

dev/data.tsv

because type is set wrong?

use the string serialization or range

@sujaypatil96 here's another one we could work on together

Right now, the MIxS environmental triad columns in the MIXS:environment field section of the NMDC Data Harmonizer accept free text (xs:token), but are validated against .* \[ENVO:\d+\]. An appropriate response for the environmental broad scale might be 'ocean [ENVO:00000015]', but the user would have to type all of that in.

Problems

• First of all, eventually we will want to accept terms form ontologies other than EnvO, like UBERON for anatomical sites like colon [UBERON:0001155]
• also, these columns could use pull-down menus, but they would be unmanageable with all terms from EnvO, much less other ontologies
  • @mslarae13 was involved in determining reasonable subsets of EnvO terms for various MIxS packages (soil, water, etc.) x each of the environmental triad columns. We could build enumerations into the LinkML that precedes the DataHarmonizer interface, which would then appear as pull-down menus.

@mslarae13 can you please remind me where to find the curated subsets?

@sujaypatil96 I can walk you though more of this if you can budget some time for this.

Triad columns

CC @cmungall

related to #31?

SheetIdentification

Tests have been started for this task

see also #59

• all tabs that appear in Soil-NMDC-Template_Compiled must be listed in the SheetIdentification tab
• the tabs should (?) be listed in the order they appear in Soil-NMDC-Template_Compiled
• tabs that are expected to be parsed in creation of DH templates must be tagged as input for soil DH template generation in SheetIdentification
• tabs that aren't tagged as to be parsed in creation of DH templates, or don't serve some other well-characterized purpose should be deleted
  • maybe we should designate some tabs as scratch space?

I need to do a better job of communicating these to @mslarae13 or agreeing on alternatives

Regarding Soil-NMDC-Template_Compiled

• tab name and ordering won't change.
  • support for adding new tabs to the left should be added.
• tab names and order will match the sheet_name column in tab SheetIdentification
  • not implemented yet
• no otherwise populated row in the Terms tab will have an empty Column Header

• if the pattern looks like a list, make it a enumeration/pulldown

  • solved by processing MIxS after NMDC because the list-like patterns came from NMDC and get overwritten by MIxS?
  • they are still in MIxS string serialization

• add hierarchical indentation of enumerated values

  • I have some old code for that somewhere
  • probably requires sem-sql code and database (vs a SPARQL solution)
  • use any ontologies besides EnvO?
  • see #58

• Add support for partial date columns and time columns

• align section composition and ordering with @mslarae13's Example Use tab

  • that info will be added to other more structured tabs

• tidy the descriptions

  • remove quotes (from NMDC, like env_local_scale. When MIxS is processed after NMDC, the quote-free MIxS annotations supersede.) See microbiomedata/nmdc-schema#229
  • remove square brackets (from NMDC, like GOLD sample identifiers) See microbiomedata/nmdc-schema#228
  • 'TODO' description on some NMDC terms. See microbiomedata/nmdc-schema#231

• add meanings for enumerated values

  • lookup with enum_annotator (see example in Makefile)
  • expose as Ontology IDs
  • some enum labels from MIxS have added parenthetic content that makes string matching difficult: meadows (grasses,alfalfa,fescue,bromegrass,timothy)

• add more patterns based on

  • string_serialization
  • slot's range... pretty thorough at this point

• populate examples column?

  • but microbiomedata/nmdc-schema#233

• are terms being included even though they are marked skip on nmdc_biosample_slots ?

• Ontology ID

  • full URLs (prefer prefixed)
    • gen-yaml NMDC and MIxS schemas, yq delete imports sections
  • MIxS slots with NMDC URLs like https://microbiomedata/schema/mixs/ph

• terse labels (from apparent prioritization of NMDC over MIxS annotations?)

• parent classes with "https:" prefixes

  • long-term solution: align section composition and ordering below
  • short-term solution comes from full URLs (prefer prefixed) above

• seems like number of required fields too low

  • added requirements from slot usages

• elaborate on the use of regular expressions in the guidance column. Also include the string serialization?

• Where is the default PV in the sample_type enum coming from

  • @click.option('--default_data_status', default="default", show_default=True)

• what does the Null values section in the double-click header help mean? see cidgoh#244

  • shows the contents of the data status column in data.tsv, which I was populating with --default_data_status

• take advantage of min and max values for pH (anything else?)

• whose id-like fields should be used? The ones from NMDC or ones created by @mslarae13

  • using identifiers from biosample_identification_slots

instead of poetry run pytest do poetry run pytest -s test_sntc.py

see also #4

Soil-NMDC-Template_Compiled is the input into this repo's pipeline for building DataHarmonizer data.tsv templates, so it can't be edited in a completely open-ended way.

Some technical or policy solution should be developed so that the author(s) (@mslarae13) of Soil-NMDC-Template_Compiled are enabled to make corrections and improvements.

Soil-NMDC-Template_Compiled

Currently, most of the methods use here just overwrite terms. We're largely trusting Soil-NMDC-Template_Compiled to not request a slot from two different sources

@cmungall , could you help me try to generalize a pattern for running LME enum annotation over several different enums, in series?

We'll probaly want to tune the ontology_string, max_cosine and trim_parentheticals for each requested_enum_name

I do remember your request to separate the annotation from the NMDC DH build

• more class orientation
• better name for linkml_round_trips folder, broken out into topical subdirectories

My fork removes some of the comments (Occurrence, slot used in packages) as those don't seem helpful in the DataHarmonizer Guidance.

see turbomam/mixs-source@78d1fe9

see also #2

default --client_secret_json parameter for sheet2linkml script should be local/client_secret.apps.googleusercontent.com.json

Failed to map:

The template for the loaded file has a configuration error:
dcterms:default
This is a field that has no parent, or a section that has no fields.

linkml_round_trips is the home of the Python code, including the X -> LinkML -> DH data.tsv conversion.

Human + API repair enrichment and curation of EMSL, JGI or KBase data?

Integration with NMDC runtime?

@hubin-keio and @scanon, do you have anything to contribute?

see https://github.com/turbomam/hident

And templates/old

They'll still be in GH somewhere!

Email from [email protected]

The page build failed with the following error:
Page build failed.
For information on troubleshooting Jekyll see:
https://docs.github.com/articles/troubleshooting-jekyll-builds

If you have any questions you can submit a request at https://support.github.com/contact?tags=dotcom-pages&repo_id=439138059&page_build_id=301228892

Add EMSL and JGI terms ASAP too

Related work

• See also #20
• make all already comes close to this but is too inclusive (and the workflow is a mess)
• See also modular_gd.py

Terms will come from Soil-NMDC-Template_Compiled

The result should be a LinkML YAML file that passes basic validation, like serving as the input into gen-yaml

Take slot terms from the mixs_packages_x_slots tab where the package column = 'soil' and the package column = 'use as-is or 'borrowed'

Take name terms from tab nmdc_biosample_slots where column from_schema != 'https://microbiomedata/schema/mixs'

All of those slots should be assigned to a new class like soil_biosample

just validate a bare float

@mslarae13

Requesting section-orients navigation in addition to column-based navigation.

Who's in the best position to implement these? Somebody in NMDC, Kitware, or CIDGOH?

Hoping to implement by early February, for a late February conference demo

• add jump to section
• add show/hide section

Traceback (most recent call last):
File "/home/mark/.cache/pypoetry/virtualenvs/linkml-round-trips-2m4fR-mZ-py3.9/bin/gen-yaml", line 8, in
sys.exit(cli())
File "/home/mark/.cache/pypoetry/virtualenvs/linkml-round-trips-2m4fR-mZ-py3.9/lib/python3.9/site-packages/click/core.py", line 829, in call
return self.main(*args, **kwargs)
File "/home/mark/.cache/pypoetry/virtualenvs/linkml-round-trips-2m4fR-mZ-py3.9/lib/python3.9/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/mark/.cache/pypoetry/virtualenvs/linkml-round-trips-2m4fR-mZ-py3.9/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/mark/.cache/pypoetry/virtualenvs/linkml-round-trips-2m4fR-mZ-py3.9/lib/python3.9/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/mark/.cache/pypoetry/virtualenvs/linkml-round-trips-2m4fR-mZ-py3.9/lib/python3.9/site-packages/linkml/generators/yamlgen.py", line 45, in cli
gen = YAMLGenerator(yamlfile, **args)
File "/home/mark/.cache/pypoetry/virtualenvs/linkml-round-trips-2m4fR-mZ-py3.9/lib/python3.9/site-packages/linkml/generators/yamlgen.py", line 24, in init
super().init(schema, **kwargs)
File "/home/mark/.cache/pypoetry/virtualenvs/linkml-round-trips-2m4fR-mZ-py3.9/lib/python3.9/site-packages/linkml/utils/generator.py", line 91, in init
loader = SchemaLoader(schema, self.base_dir, useuris=useuris, importmap=importmap, logger=self.logger,
File "/home/mark/.cache/pypoetry/virtualenvs/linkml-round-trips-2m4fR-mZ-py3.9/lib/python3.9/site-packages/linkml/utils/schemaloader.py", line 52, in init
self.schema = load_raw_schema(data, base_dir=base_dir, merge_modules=mergeimports,
File "/home/mark/.cache/pypoetry/virtualenvs/linkml-round-trips-2m4fR-mZ-py3.9/lib/python3.9/site-packages/linkml/utils/rawloader.py", line 69, in load_raw_schema
schema = yaml_loader.load(copy.deepcopy(data) if isinstance(data, dict) else data,
File "/home/mark/.cache/pypoetry/virtualenvs/linkml-round-trips-2m4fR-mZ-py3.9/lib/python3.9/site-packages/linkml_runtime/loaders/yaml_loader.py", line 22, in load
return self.load_source(source, loader, target_class, accept_header="text/yaml, application/yaml;q=0.9",
File "/home/mark/.cache/pypoetry/virtualenvs/linkml-round-trips-2m4fR-mZ-py3.9/lib/python3.9/site-packages/linkml_runtime/loaders/loader_root.py", line 60, in load_source
data_as_dict = loader(data, metadata)
File "/home/mark/.cache/pypoetry/virtualenvs/linkml-round-trips-2m4fR-mZ-py3.9/lib/python3.9/site-packages/linkml_runtime/loaders/yaml_loader.py", line 16, in loader
return yaml.load(StringIO(data), DupCheckYamlLoader) if isinstance(data, str) else data
File "/home/mark/.cache/pypoetry/virtualenvs/linkml-round-trips-2m4fR-mZ-py3.9/lib/python3.9/site-packages/yaml/init.py", line 114, in load
return loader.get_single_data()
File "/home/mark/.cache/pypoetry/virtualenvs/linkml-round-trips-2m4fR-mZ-py3.9/lib/python3.9/site-packages/yaml/constructor.py", line 49, in get_single_data
node = self.get_single_node()
File "/home/mark/.cache/pypoetry/virtualenvs/linkml-round-trips-2m4fR-mZ-py3.9/lib/python3.9/site-packages/yaml/composer.py", line 36, in get_single_node
document = self.compose_document()
File "/home/mark/.cache/pypoetry/virtualenvs/linkml-round-trips-2m4fR-mZ-py3.9/lib/python3.9/site-packages/yaml/composer.py", line 58, in compose_document
self.get_event()
File "/home/mark/.cache/pypoetry/virtualenvs/linkml-round-trips-2m4fR-mZ-py3.9/lib/python3.9/site-packages/yaml/parser.py", line 118, in get_event
self.current_event = self.state()
File "/home/mark/.cache/pypoetry/virtualenvs/linkml-round-trips-2m4fR-mZ-py3.9/lib/python3.9/site-packages/yaml/parser.py", line 193, in parse_document_end
token = self.peek_token()
File "/home/mark/.cache/pypoetry/virtualenvs/linkml-round-trips-2m4fR-mZ-py3.9/lib/python3.9/site-packages/yaml/scanner.py", line 129, in peek_token
self.fetch_more_tokens()
File "/home/mark/.cache/pypoetry/virtualenvs/linkml-round-trips-2m4fR-mZ-py3.9/lib/python3.9/site-packages/yaml/scanner.py", line 223, in fetch_more_tokens
return self.fetch_value()
File "/home/mark/.cache/pypoetry/virtualenvs/linkml-round-trips-2m4fR-mZ-py3.9/lib/python3.9/site-packages/yaml/scanner.py", line 577, in fetch_value
raise ScannerError(None, None,
yaml.scanner.ScannerError: mapping values are not allowed here

Authentication is required before these scripts (and tests) can access the Soil-NMDC-Template_Compiled Google Sheet. Documentation is required.

This is especially tricky if running tests like test_sntc.py if authentication is incomplete. If local/client_secret.apps.googleusercontent.com.json is present but sheets.googleapis.com-python.json isn't, run it like this the first time around:

poetry run pytest -s test_sntc.py

You will be asked to open a URL, grant authorization to these scripts, and then paste a code back into the shell

dependent on arguments to linkml to DH script

samp_id:unique_ID and samp_id:sample_link require pattern [^\:\n\r]+\:[^\:\n\r]+