Giter VIP home page Giter VIP logo

geocodes_documentation's Introduction

geocodes_documentation's People

Contributors

mbcode avatar smrgeoinfo avatar valentinedwv avatar ylyangtw avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

geocodes_documentation's Issues

Data load validation for data loading

  • What do we need to validate a data load
  • (Gleaner/nabu) logs, and scripts to look for errors
  • Notebook to Validate Dataloading for ECO Staff
    • Duplicate checking
    • Loaded counts

Specify Logic of Data Loading and validation

There is a validation notebook prototypes
There are site map, sparql, s3 and other counts shown

Put the existing and future logic into a document. This will be the basis for discussing the what parts of the validation are implemented, and how we test them.
It it's a living document (aka notebook), please add full descriptions of the steps, then implement.
Aka test description first, code second.

Outline of User Tutorial

This should be based on the onbording a source document.
Ideally people attending would have a repository/organization.

Document Data Loading Validation Logic

ADD TO THIS DOCUMENT: https://github.com/earthcube/geocodes_documentation/wiki/DataLoadingValidationStory#what-do-we-need-do-to-setup-testing

Document the testing that will be needed to validate the data loading.
Repeat, document the steps we will need to implement to validate the data loading.

This is not asking for the implementation of the tests. It is asking for the testing plan with the steps that will will need to implement.

Places to possibly look:

Examples

Spec: Summon working

  • count of records from a sitemap (matches) count of records that made it into the bucket
  • possible tool: sitemap assay

Spec: expected JSON-LD
possible tooling: https://github.com/gleanerio/notebooks/tree/master/notebooks/validation

  • validate jsonld using ?
  • does it have xxxx

Spec: JSON-LD data load to triple store

  • count of files in s3 bucket (matches) count of graphs for an organization/record
  • For (% of data load or all if < 100 records)
    • retrieve JSONLD
      • urn
      • name
    • retrieve a graph by a urn from the service api
      • did we get a record
      • does information (name, urn, [other properties]) match
    • retrieve from triplestore using user interface query with name as

Spec: JSON LD Renders in UI

  • for a random number of jsonld in org.
    • create a set of curl url
    • open in a web test suite (selenium, etc) and see that they at least partially render to expectations

Spec Tool Linkage:

  • for a set of known files, do we match the look linkages.
    • This probably needs to be an approval test.

spec: org provenance information

System Overview Documentation

Over the next few weeks onboard Luigi and Ya-Lan
Create systems overview documentation.
Let's put in the geocodes_documentation repository.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.