We currently hard-code ACMG criteria for TSV import. This should be configurable per profile.

Implement mypy type checking in the CI.

We should document how things work internally and fit in with the ClinVar API, explain files, config settings etc.

Can currently only be set on import.

There are a a few lines not covered by the tests. Write tests that cover them.

Is your feature request related to a problem? Please describe.
In many situations it is useful to see the links between HPO terms and genes from ClinVar.

Describe the solution you'd like
Now that we have the handy JSONL format, generating this is easy. A similar approach as in #102 can be taken.

Describe alternatives you've considered
N/A

Additional context
N/A

Implement CLI for

• clinvar-this config get NAME
• clinvar-this config set NAME VALUE

At the moment, it releases and tries to push snakemake.

Is your feature request related to a problem? Please describe.
Lollipop plots are a useful visualization of variants in a gene. See for example the ClinVar ones. They would be useful to have in the output.

Describe the solution you'd like
Besides the coordinates of the transcript / exon (which would have to come from Mehari), we would need to write out (separate for each genome release) for each gene, all variant:

• SPDI information (chrom, pos, deleted bases, inserted bases)
• molecular impact
• RCV
• ACMG rating
• review status

Describe alternatives you've considered
N/A

Additional context
N/A

Implement a clinvar_api.client module that provides access to the NCBI ClinVar REST API.

Is your feature request related to a problem? Please describe.
For annonars and display we need to have files with small and structural variants.

Describe the solution you'd like
Write out such files that contain the variant, the RCV, and the assessment and review status.

Describe alternatives you've considered
N/A

Additional context
N/A

Improve support for profiles and write share data into profile sub directory.

Add models for parsing the summary response messages by the schema, as given in the ncbi docs.

Add unit tests for parsing submission message classes.

Store as extra_data so it can be exported to TSV.

A number of enumerations are still strings. They should be enums.

Is your feature request related to a problem? Please describe.
When working with ClinVar, it is useful to work with their full database dumps published in XML format. However, the XML format is hard to work with.

Describe the solution you'd like
Add functionality to parse the ClinVar data based on xmltodict and further processing to easy-to-use Python attrs classes. Importantly, allow export to JSONL format as this is easy to work with in external tools as wells.

Describe alternatives you've considered
N/A

Additional context
N/A

Is your feature request related to a problem? Please describe.
In the case that more than one variant is submitted for a patient, we currently have no way of detecting this.

Describe the solution you'd like
Write out the submitting organisation and local key (when available) for each record in the phenotype to gene link step.

Describe alternatives you've considered
N/A

Additional context
N/A

Add unit tests for submission messages.

Phenopackets is an important data exchange format. Support import/export for it from the internal JSON data structures.

Currently this is missing.

Create a simple TSV format that can be used to create submissions.

Create an implementation of models for server communication based on attrs.

These are pretty common so it would be nice to be able to import them.

While #6 specifies the models for the messages we should have something for the internal API to use with snake_case etc.

This is currently not possible yet.

Use a fuzzing library to generate different JSON input and pass it through the parser. Make sure to use a fixed seed.

While #8 has the message models, we should have internal models with snake_case for the submission.

Document the following use cases:

• import from various formats into the internal JSON format
• submission and result retrieval
• submission and resubmission
• deletion of records

Email from ClinVar

Dear ClinVar API Submitter,

Thank you for using the ClinVar Submission API!
We are making changes to the API submission "test mode" to allow users to test the new schema before its production release.
The test endpoint for testing submission of data https://submit.ncbi.nlm.nih.gov/apitest/v1/submissions could be unstable during the transition to this new test schema from November 11-13, 2022.
From November 14, 2022, it will only accept the requests adhering to the new upcoming schema.
Key changes in new submission schema:

• We have moved 'assertionCriteria' to top level. Note that only one single assertion criteria (a citation or an electronic document) may be provided for a submission. This assertion criteria will be applied to all interpretations in the submission.
• We have moved 'releaseStatus' to the top level as 'clinvarSubmissionReleaseStatus'. This status will be applied to all the interpretations in the submission.

Please see the new submission schema attached for details.

Note: The main API enpoint https://submit.ncbi.nlm.nih.gov/api/v1/submissions/ still accepts the current schema at https://www.ncbi.nlm.nih.gov/clinvar/docs/api_http/#h2_schema_raw.
We will do the production release of this new schema on the main endpoint on November 21, 2022.
If you have any questions, please contact us at [email protected].

Thank you for sharing data through ClinVar!

ClinVar staff
[email protected]

ClinVar API Error Message

Additional properties are not allowed ('assertionCriteria', 'releaseStatus' were unexpected)

This is currently missing yet.

• clinical_significance.date_last_evaluated
• clinical_significance.comment

Currently, we overwrite. We should not do this.

Implement enumerations from API models as Python enums

• add sphinx-based documentation
• create corresponding project on readthedocs

Is your feature request related to a problem? Please describe.
Having the ClinVar XML parsing features is useful but this has not been reflected in the documentation yet.

Describe the solution you'd like
Augment the documentation both in terms of manual and API docs.

Describe alternatives you've considered
N/A

Additional context
N/A

We need more Python sphinx docstrings for members of the messages etc. from the ClinVar API documentation.

Is your feature request related to a problem? Please describe.
Having the JSONL files is handy as now it will be easy to create a per-gene report of how many variants with each molecular impact and each clinvar pathogenicity are present.

Describe the solution you'd like
Create a clinvar-this data gene-impact-report command for this.

Describe alternatives you've considered
N/A

Additional context

• The aim is to support the upper part of the image here: varfish-org/annonars#154

We do not use this anywhere. Remove from CLI, internals, and docs.

Implement attrs based models for creating submission messages by the docs.

Currently we only allow submission of isolated variants.

At the moment, it is translated into OMIM ID not provided.

Is your feature request related to a problem? Please describe.
It can be useful to know the distribution of variants classified as ACMG1-5 for each gene.

Describe the solution you'd like
Create a command that reports for each gene the number of variants in each ACMG classification in buckets

• 0 / no frequency
• <0.00001
• <0.000025
• <0.00005
• <0.0001
• <0.00025
• <0.0005
• <0.001
• <0.0025
• <0.005
• <0.01
• <0.025
• <0.05
• <0.1
• <0.25
• <0.5
• <1

Describe alternatives you've considered
N/A

Additional context
N/A

We need to add models for the extra files, e.g.

{
  "submissionName": "SUB673156",
  "submissionDate": "2021-03-25",
  "batchProcessingStatus": "Success",
  "batchReleaseStatus": "Not released",
  "totalCount": 3,
  "totalErrors": 0,
  "totalSuccess": 3,
  "totalPublic": 0,
  "submissions": [
    {
      "identifiers": {
        "localID": "adefc5ed-7d59-4119-8b3d-07dcdc504c09_success1",
        "clinvarLocalKey": "adefc5ed-7d59-4119-8b3d-07dcdc504c09_success1",
        "localKey": "adefc5ed-7d59-4119-8b3d-07dcdc504c09_success1",
        "clinvarAccession": "SCV000839746"
      },
      "processingStatus": "Success"
    },
    {
      "identifiers": {
        "localID": "adefc5ed-7d59-4119-8b3d-07dcdc504c09_success2",
        "clinvarLocalKey": "adefc5ed-7d59-4119-8b3d-07dcdc504c09_success2",
        "localKey": "adefc5ed-7d59-4119-8b3d-07dcdc504c09_success2",
        "clinvarAccession": "SCV000839747"
      },
      "processingStatus": "Success"
    },
    {
      "identifiers": {
        "localID": "adefc5ed-7d59-4119-8b3d-07dcdc504c09_success3",
        "clinvarLocalKey": "adefc5ed-7d59-4119-8b3d-07dcdc504c09_success3",
        "localKey": "adefc5ed-7d59-4119-8b3d-07dcdc504c09_success3",
        "clinvarAccession": "SCV000839748"
      },
      "processingStatus": "Success"
    }
  ]
}

Document getting started, beginning with registering an org, getting an API key, importing data using the apitest and dry-run, and finally removing the records again.

The message records from #1 are to be used for server communication. The aim of this ticket is to abstract away the length-1 arrays and lowerCamelCase to snake_case.

Currently, extra columns are ignored in TSV import. We should store them in an attribute extra_data in the internal models.