podaac / hydrocron Goto Github PK

View Code? Open in Web Editor NEW

11.0 8.0 2.0 47.11 MB

API for retreiving time series of SWOT data

Home Page: https://podaac.github.io/hydrocron/

License: Apache License 2.0

Python 76.86% Dockerfile 0.94% Shell 2.25% HCL 19.95%

development tva

hydrocron's Introduction

Overview

Hydrocron API is a new tool that implements functionalities that will allow hydrologists to have direct access to filtered data from our newest satellites. This innovative tool will provide an effortless way to filter data by feature ID, date range, polygonal area, and more. This data will be returned in formats such as CSV and geoJSON.

Requirements

Python 3.10+

Running Locally with Docker

Build or pull the hydrocron docker image
Run docker compose to launch dynamodb local and hydrocron local
Load test data into dynamodb local
Execute sample requests

1. Build or Pull Hydrocron Docker

Build the docker container:

docker build . -f docker/Dockerfile -t hydrocron:latest

Pull a pre-built image from https://github.com/podaac/hydrocron/pkgs/container/hydrocron:

docker pull ghcr.io/podaac/hydrocron:latest

2. Run Docker Compose

Launch dynamodb local on port 8000 and hyrdrocron on port 9000

docker-compose up

3. Load Test Data

If you have not setup a python environment yet, use poetry to first initialize the virtual environment.

poetry install

This will load the data in test/data into the local dynamo db instance.

python tests/load_data_local.py

NOTE - By default data will be removed when the container is stopped. There are some commented lines in docker-compose.yml that can be used to allow the data to persist across container restarts if desired.

4. Execute Sample Requests

The docker container is running a lambda container image. By posting data to port 9000, the lambda handler will be invoked and will return results from the loaded test data. For example:

curl --location 'http://localhost:9000/2015-03-31/functions/function/invocations' \
--header 'Content-Type: application/json' \
--data '{
    "body":{
        "feature": "Reach",
        "reach_id": "71224100223",
        "start_time": "2022-08-04T00:00:00+00:00",
        "end_time": "2022-08-23T00:00:00+00:00",
        "output": "csv",
        "fields": "feature_id,time_str,wse"
    }
}'

Loading the Database from CMR

Instructions for loading the database with data from CMR instead of the single test granule, are described in hydrocron_db/LOAD_DATA_README.md

hydrocron's People

Contributors

Stargazers

Watchers

Forkers

cuahsi gardnerlab-pitt

hydrocron's Issues

Remove flask app

Will be using API Gateway without swagger-ui, no need to start flask app.

Implement small buffer around time in queries

All time fields (time, time_str, time_tai) in SWOT hydrology source shapefiles can contain 'no_data' or fillvalues, since these represent the 'time of observation' and there are circumstances under which an observation cannot be recorded for a given reach, node, or lake. To handle this, hydrocron uses the continent pass range start time as the sort key in the database and the time to query on, rather than the observed time.

This means that when a user enters a query with a start_time of 2023-06-01T12:00:00 and an end_time of 2023-12-01T12:00:00, hydrocron will return only records that have a range_start_time within that time range. However, it's possible that a feature was observed at 2023-06-01T12:00:05, but the continent pass started at 2023-06-01T11:59:55. In this case the range_start_time = 2023-06-01T11:59:55 and the feature that was observed within the time bounds of the query won't be included in the results.

Implementing a buffer of 30 seconds before the start_time the user enters will ensure that all observations within the time range the user requests are returned. Note that in some cases this will return features with an observed time that is prior to the start_time entered, but given the repeat time of observations is 21 days, this should occur rarely and will return at most one extra timestep that is within 30s of the time requested.

Remove partial_f from fields to obscure

It's often causing data loading failures

Request EDC Metrics integration

Would like to ensure hydrocron metrics are available via https://metrics.earthdata.nasa.gov/

Using GeoDataFrame to work with data returned from db

We may be able to simplify the API code if we convert the data returned from the dynamodb queries back into a geodataframe and use geopandas for the formatting:

Load data lambda only loads the first granule found in the time range

Lmabda function successfully loads the first granule found in the CMR query and then fails with the following message:
Calling the invoke API action failed with this message: Failed to fetch

Solve issue with "shape" library (Docker container)

We expect this to be solved by #7

Complete CI/CD Pipeline

Pipeline currently stops at pytest. Need to implement remaining steps through deployment to AWS

Create dashboard for monitoring

TBD: What should the dashboard show?

number of requests
size of data transferred
avg. request time
Error rate
IP geolocation of request origin

update API documentation with query response examples

Cut release for 1.0.0

DiT: Use VPC endpoint for DynamoDB access

https://wiki.earthdata.nasa.gov/pages/viewpage.action?spaceKey=ENS&title=PI+24.1+Tenant+Action+Items

Create API Usage Documentation

End user documentation for how to use/integrate the API into their applications.
Use sphinx for building documentation
Published to github pages

Integrate with GH Actions (stretch, may break this out into another issue)

Change database query to use time range

The query key condition expression should only return items within the date/time range entered. Currently the API returns all data for a given feature ID regardless of start/end time.


items = hydrocron_table.query(KeyConditionExpression=Key(
    constants.SWOT_REACH_PARTITION_KEY).eq(feature_id))

Could not parse request body into json

Report from NGAP during setup of the cloudfront distribution. When sending a request they were receiving a json error:

{“message”: “Could not parse request body into json: Could not parse payload into json: Unexpected character (\‘\“\’ (code 34)): was expecting comma to separate Object entries\n at [Source: (byte[])\“{\n\“body\” : {\n            \“end_time\” : \“2023-10-30T00:00:00Z\“\n        ,        \“feature\” : \“Reach\“\n        ,        \“feature_id\” : \“71224301163\“\n        ,        \“fields\” : \“reach_id,time_str,wse,geometry\“\n        ,        \“output\” : \“csv\“\n        ,        \“start_time\” : \“2023-06-01T00:00:00Z\“\n                    \“Accept\” : \“text\\\/html,application\\\/xhtml+xml,application\\\/xml;q=0.9,image\\\/avif,image\\\/webp,*\\\/*;q=0.8\“\n        ,        \“Accept-Encoding\” : \“gzip\“\n        ,        \“Host\” : \“i930rqhz2m\“[truncated 1170 bytes]; line: 9, column: 22]“}

Include Terraform in this merge

For simplicity, new ticket just to include Terraform (merged from hydrocron_api and hydrocron_db repos)

Integrate API documentation with github actions

Structure of object returned is different between csv and geojson output

CSV output is structured like:

{
    "status": "200 OK",
    "results": "*STRING OF CSV CONTENT*"
}

geojson output is structured like:

{
    "status": "200 OK",
    "results": {
        "status": "200 OK",
        "time": "219.917 ms.",
        "type": "FeatureCollection",
        "features": [{*GEOJSON FEATURE OBJECT*}],
        "hits": 3
    }
}

The structure of the output should be the same for both format types. I would suggest something like

{
    "status": "200 OK",
    "time": "219.917 ms.",
    "hits": 3
    "results": {
        "csv": "*CSV FORMATTED DATA IF CSV REQUESTED, EMPTY STRING OTHERWISE*",
        "geojson": *DATA AS GEOJSON FEATURECOLLECITON IF GEOJSON REQUESTED, EMPTY OBJECT OTHERWISE*
    }
}

Move all constants to separate constants file

Table names, column names, etc should all be moved to contants file instead of hard coded as strings throughout.

Improve error handling

Problem:

When sending a request that is missing required query parameters, a 500 error is thrown.

Example:

https://soto.podaac.sit.earthdatacloud.nasa.gov/hydrocron/v1/timeseries

Result:

< HTTP/2 500 
< content-type: application/json
< content-length: 36

{"message": "Internal server error"}

Expected behavior:

Return a 400 Bad Request response indicating which required query parameters were missing from the request.

CSV output incorrect when requesting geometry

Problem:

When including the geometry field for csv results, the returned output does not contain the data for the other fields

Request:

timeseries?feature=Reach&reach_id=71224301163&start_time=2023-06-01T00:00:00Z&end_time=2023-10-30T00:00:00Z&output=csv&fields=reach_id,time_str,wse,geometry

Result:

{"status":"200 OK","time":2055.722,"hits":3,"results":{"csv":"reach_id,time_str,wse,geometry\nLINESTRING (-95.200201 49.100687, -95.200295 49.100955, -95.200471 49.101221, -95.200565 49.101489, -95.200659 49.101757, -95.200754 49.102026, -95.200848 49.102294, -95.200942 49.102562, -95.201036 49.10283, -95.20113 49.103098, -95.201224 49.103366, -95.2014 49.103633, -95.201494 49.103901, -95.201588 49.104169, -95.201682 49.104437, -95.201859 49.104703, -95.201953 49.104971, -95.202047 49.10524, -95.202141 49.105508, -95.202317 49.105774, -95.202411 49.106042, -95.202505 49.10631, -95.202599 49.106578, -95.202693 49.106847, -95.202788 49.107115, -95.202882 49.107383, -95.202976 49.107651, -95.20307 49.107919, -95.203246 49.108186, -95.20334 49.108454, -95.203434 49.108722, -95.203528 49.10899, -95.203705 49.109256, -95.203799 49.109524, -95.203893 49.109793, -95.203987 49.110061, -95.204081 49.110329, -95.204175 49.110597, -95.204269 49.110865, -95.204363 49.111133, -95.204457 49.111401, -95.204634 49.111668, -95.204728 49.111936, -95.204822 49.112204, -95.204916 49.112472, -95.205092 49.112738, -95.205186 49.113006, -95.20528 49.113275, -95.205375 49.113543, -95.205469 49.113811, -95.205563 49.114079, -95.205657 49.114347, -95.205751 49.114615, -95.205845 49.114883, -95.206021 49.11515, -95.206116 49.115418, -95.20621 49.115686, -95.206304 49.115954, -95.20648 49.11622, -95.206574 49.116489, -95.206668 49.116757, -95.206763 49.117025, -95.206857 49.117293, -95.206951 49.117561, -95.207045 49.117829, -95.207139 49.118097, -95.207233 49.118365, -95.207327 49.118633, -95.207339 49.118903, -95.207351 49.119173, -95.207363 49.119442, -95.207375 49.119712, -95.207387 49.119982, -95.207399 49.120251, -95.207329 49.120522, -95.207259 49.120794, -95.207189 49.121065, -95.207036 49.121338, -95.206884 49.121611, -95.206814 49.121882, -95.206661 49.122155, -95.206509 49.122427, -95.206439 49.122699, -95.206286 49.122971, -95.206134 49.123244, -95.206064 49.123515, -95.205911 49.123788, -95.205759 49.124061, -95.205689 49.124332, -95.205619 49.124604, -95.205466 49.124876, -95.205396 49.125148, -95.205326 49.125419, -95.205173 49.125692, -95.205021 49.125964, -95.204951 49.126236, -95.204798 49.126508, -95.204646 49.126781, -95.204576 49.127053, -95.204423 49.127325, -95.204271 49.127598, -95.204201 49.127869, -95.204048 49.128142, -95.203896 49.128415, -95.203826 49.128686, -95.203673 49.128959, -95.203521 49.129232, -95.203451 49.129503, -95.203298 49.129776, -95.203146 49.130049, -95.203076 49.13032, -95.202923 49.130593, -95.202771 49.130865, -95.202701 49.131137, -95.202548 49.131409, -95.202396 49.131682, -95.202325 49.131954, -95.202173 49.132226, -95.202021 49.132499, -95.20195 49.13277, -95.201798 49.133043, -95.201645 49.133316, -95.201575 49.133587, -95.201423 49.13386, -95.20127 49.134133, -95.201118 49.134406, -95.200965 49.134678, -95.200813 49.134951, -95.200661 49.135224, -95.200508 49.135497, -95.200438 49.135768, -95.200285 49.136041, -95.200133 49.136314, -95.200063 49.136585, -95.19991 49.136858, -95.199758 49.13713, -95.199688 49.137402, -95.199535 49.137674, -95.199383 49.137947, -95.199312 49.138218, -95.19916 49.138491, -95.199007 49.138764, -95.198937 49.139035, -95.198785 49.139308, -95.198632 49.139581, -95.198562 49.139852, -95.19841 49.140125, -95.198257 49.140398, -95.198187 49.140669, -95.198034 49.140942, -95.197882 49.141214, -95.197812 49.141486, -95.197659 49.141759, -95.197507 49.142031, -95.197436 49.142303, -95.197284 49.142575, -95.197131 49.142848, -95.197061 49.143119, -95.196909 49.143392, -95.196756 49.143665, -95.196604 49.143938, -95.196451 49.144211, -95.196216 49.144485, -95.196064 49.144758, -95.195829 49.145032, -95.195595 49.145306, -95.19536 49.145581, -95.195125 49.145855, -95.194808 49.146131, -95.194573 49.146405, -95.194339 49.14668, -95.194022 49.146956, -95.193705 49.147232, -95.19347 49.147506, -95.193153 49.147782, -95.192836 49.148058, -95.192519 49.148334, -95.192202 49.14861, -95.191885 49.148885, -95.191568 49.149161, -95.191251 49.149437, -95.190934 49.149713, -95.190617 49.149989, -95.190218 49.150267, -95.189819 49.150544, -95.18942 49.150821, -95.189021 49.151099, -95.188621 49.151376, -95.188222 49.151654, -95.187823 49.151931, -95.187424 49.152209, -95.187024 49.152486, -95.186625 49.152764, -95.186226 49.153041, -95.185827 49.153319, -95.185425 49.153542, -95.185023 49.153766, -95.184622 49.153989, -95.18422 49.154213, -95.183819 49.154436, -95.183419 49.154714, -95.183018 49.154937, -95.182616 49.155161, -95.182214 49.155384, -95.18181 49.155554, -95.181406 49.155723, -95.181002 49.155893, -95.180598 49.156062, -95.180194 49.156232, -95.17979 49.156402, -95.179386 49.156571, -95.178982 49.156741, -95.178578 49.15691, -95.178171 49.157026, -95.177767 49.157195, -95.177361 49.157311, -95.176955 49.157427, -95.176546 49.157488, -95.176139 49.157604, -95.175731 49.157666, -95.175322 49.157727, -95.174913 49.157789, -95.174506 49.157905, -95.174098 49.157966, -95.173689 49.158028, -95.173282 49.158144, -95.172876 49.158259, -95.172467 49.158321, -95.172061 49.158436, -95.171654 49.158552, -95.171245 49.158614, -95.170839 49.158729, -95.170433 49.158845, -95.170024 49.158907, -95.169617 49.159022, -95.169211 49.159138, -95.168802 49.159199, -95.168396 49.159315, -95.167989 49.159431, -95.16758 49.159492, -95.167174 49.159608, -95.166767 49.159723, -95.166358 49.159785, -95.165952 49.159901, -95.165546 49.160016, -95.165137 49.160078, -95.16473 49.160194, -95.164324 49.160309, -95.163915 49.160371, -95.163506 49.160432, -95.1631 49.160548, -95.162691 49.16061, -95.162282 49.160671, -95.161873 49.160733, -95.161464 49.160795, -95.161055 49.160856, -95.160647 49.160918, -95.160238 49.160979, -95.159829 49.161041, -95.15942 49.161103, -95.159011 49.161164, -95.158602 49.161226, -95.158193 49.161288, -95.157785 49.161349, -95.157376 49.161411, -95.156967 49.161472, -95.156558 49.161534, -95.156149 49.161596, -95.15574 49.161657, -95.155331 49.161719, -95.154923 49.16178, -95.154514 49.161842, -95.154105 49.161904, -95.153696 49.161965, -95.153287 49.162027, -95.152878 49.162089, -95.152469 49.16215, -95.152061 49.162212, -95.151652 49.162273, -95.151243 49.162335, -95.150834 49.162397, -95.150423 49.162404, -95.150009 49.162358, -95.149596 49.162312, -95.14918 49.162211, -95.148759 49.162003, -95.148341 49.161849, -95.147923 49.161695, -95.147505 49.161541, -95.147086 49.161387, -95.146668 49.161233, -95.14625 49.161078, -95.145832 49.160924, -95.145411 49.160716, -95.144993 49.160562, -95.144575 49.160408, -95.144157 49.160254, -95.143739 49.1601, -95.14332 49.159945, -95.142902 49.159791, -95.142484 49.159637, -95.142068 49.159537, -95.141655 49.15949, -95.141244 49.159498, -95.140832 49.159506, -95.140423 49.159567, -95.140012 49.159575, -95.139601 49.159583, -95.13919 49.15959),\nLINESTRING (-95.200201 49.100687, -95.200295 49.100955, -95.200471 49.101221, -95.200565 49.101489, -95.200659 49.101757, -95.200754 49.102026, -95.200848 49.102294, -95.200942 49.102562, -95.201036 49.10283, -95.20113 49.103098, -95.201224 49.103366, -95.2014 49.103633, -95.201494 49.103901, -95.201588 49.104169, -95.201682 49.104437, -95.201859 49.104703, -95.201953 49.104971, -95.202047 49.10524, -95.202141 49.105508, -95.202317 49.105774, -95.202411 49.106042, -95.202505 49.10631, -95.202599 49.106578, -95.202693 49.106847, -95.202788 49.107115, -95.202882 49.107383, -95.202976 49.107651, -95.20307 49.107919, -95.203246 49.108186, -95.20334 49.108454, -95.203434 49.108722, -95.203528 49.10899, -95.203705 49.109256, -95.203799 49.109524, -95.203893 49.109793, -95.203987 49.110061, -95.204081 49.110329, -95.204175 49.110597, -95.204269 49.110865, -95.204363 49.111133, -95.204457 49.111401, -95.204634 49.111668, -95.204728 49.111936, -95.204822 49.112204, -95.204916 49.112472, -95.205092 49.112738, -95.205186 49.113006, -95.20528 49.113275, -95.205375 49.113543, -95.205469 49.113811, -95.205563 49.114079, -95.205657 49.114347, -95.205751 49.114615, -95.205845 49.114883, -95.206021 49.11515, -95.206116 49.115418, -95.20621 49.115686, -95.206304 49.115954, -95.20648 49.11622, -95.206574 49.116489, -95.206668 49.116757, -95.206763 49.117025, -95.206857 49.117293, -95.206951 49.117561, -95.207045 49.117829, -95.207139 49.118097, -95.207233 49.118365, -95.207327 49.118633, -95.207339 49.118903, -95.207351 49.119173, -95.207363 49.119442, -95.207375 49.119712, -95.207387 49.119982, -95.207399 49.120251, -95.207329 49.120522, -95.207259 49.120794, -95.207189 49.121065, -95.207036 49.121338, -95.206884 49.121611, -95.206814 49.121882, -95.206661 49.122155, -95.206509 49.122427, -95.206439 49.122699, -95.206286 49.122971, -95.206134 49.123244, -95.206064 49.123515, -95.205911 49.123788, -95.205759 49.124061, -95.205689 49.124332, -95.205619 49.124604, -95.205466 49.124876, -95.205396 49.125148, -95.205326 49.125419, -95.205173 49.125692, -95.205021 49.125964, -95.204951 49.126236, -95.204798 49.126508, -95.204646 49.126781, -95.204576 49.127053, -95.204423 49.127325, -95.204271 49.127598, -95.204201 49.127869, -95.204048 49.128142, -95.203896 49.128415, -95.203826 49.128686, -95.203673 49.128959, -95.203521 49.129232, -95.203451 49.129503, -95.203298 49.129776, -95.203146 49.130049, -95.203076 49.13032, -95.202923 49.130593, -95.202771 49.130865, -95.202701 49.131137, -95.202548 49.131409, -95.202396 49.131682, -95.202325 49.131954, -95.202173 49.132226, -95.202021 49.132499, -95.20195 49.13277, -95.201798 49.133043, -95.201645 49.133316, -95.201575 49.133587, -95.201423 49.13386, -95.20127 49.134133, -95.201118 49.134406, -95.200965 49.134678, -95.200813 49.134951, -95.200661 49.135224, -95.200508 49.135497, -95.200438 49.135768, -95.200285 49.136041, -95.200133 49.136314, -95.200063 49.136585, -95.19991 49.136858, -95.199758 49.13713, -95.199688 49.137402, -95.199535 49.137674, -95.199383 49.137947, -95.199312 49.138218, -95.19916 49.138491, -95.199007 49.138764, -95.198937 49.139035, -95.198785 49.139308, -95.198632 49.139581, -95.198562 49.139852, -95.19841 49.140125, -95.198257 49.140398, -95.198187 49.140669, -95.198034 49.140942, -95.197882 49.141214, -95.197812 49.141486, -95.197659 49.141759, -95.197507 49.142031, -95.197436 49.142303, -95.197284 49.142575, -95.197131 49.142848, -95.197061 49.143119, -95.196909 49.143392, -95.196756 49.143665, -95.196604 49.143938, -95.196451 49.144211, -95.196216 49.144485, -95.196064 49.144758, -95.195829 49.145032, -95.195595 49.145306, -95.19536 49.145581, -95.195125 49.145855, -95.194808 49.146131, -95.194573 49.146405, -95.194339 49.14668, -95.194022 49.146956, -95.193705 49.147232, -95.19347 49.147506, -95.193153 49.147782, -95.192836 49.148058, -95.192519 49.148334, -95.192202 49.14861, -95.191885 49.148885, -95.191568 49.149161, -95.191251 49.149437, -95.190934 49.149713, -95.190617 49.149989, -95.190218 49.150267, -95.189819 49.150544, -95.18942 49.150821, -95.189021 49.151099, -95.188621 49.151376, -95.188222 49.151654, -95.187823 49.151931, -95.187424 49.152209, -95.187024 49.152486, -95.186625 49.152764, -95.186226 49.153041, -95.185827 49.153319, -95.185425 49.153542, -95.185023 49.153766, -95.184622 49.153989, -95.18422 49.154213, -95.183819 49.154436, -95.183419 49.154714, -95.183018 49.154937, -95.182616 49.155161, -95.182214 49.155384, -95.18181 49.155554, -95.181406 49.155723, -95.181002 49.155893, -95.180598 49.156062, -95.180194 49.156232, -95.17979 49.156402, -95.179386 49.156571, -95.178982 49.156741, -95.178578 49.15691, -95.178171 49.157026, -95.177767 49.157195, -95.177361 49.157311, -95.176955 49.157427, -95.176546 49.157488, -95.176139 49.157604, -95.175731 49.157666, -95.175322 49.157727, -95.174913 49.157789, -95.174506 49.157905, -95.174098 49.157966, -95.173689 49.158028, -95.173282 49.158144, -95.172876 49.158259, -95.172467 49.158321, -95.172061 49.158436, -95.171654 49.158552, -95.171245 49.158614, -95.170839 49.158729, -95.170433 49.158845, -95.170024 49.158907, -95.169617 49.159022, -95.169211 49.159138, -95.168802 49.159199, -95.168396 49.159315, -95.167989 49.159431, -95.16758 49.159492, -95.167174 49.159608, -95.166767 49.159723, -95.166358 49.159785, -95.165952 49.159901, -95.165546 49.160016, -95.165137 49.160078, -95.16473 49.160194, -95.164324 49.160309, -95.163915 49.160371, -95.163506 49.160432, -95.1631 49.160548, -95.162691 49.16061, -95.162282 49.160671, -95.161873 49.160733, -95.161464 49.160795, -95.161055 49.160856, -95.160647 49.160918, -95.160238 49.160979, -95.159829 49.161041, -95.15942 49.161103, -95.159011 49.161164, -95.158602 49.161226, -95.158193 49.161288, -95.157785 49.161349, -95.157376 49.161411, -95.156967 49.161472, -95.156558 49.161534, -95.156149 49.161596, -95.15574 49.161657, -95.155331 49.161719, -95.154923 49.16178, -95.154514 49.161842, -95.154105 49.161904, -95.153696 49.161965, -95.153287 49.162027, -95.152878 49.162089, -95.152469 49.16215, -95.152061 49.162212, -95.151652 49.162273, -95.151243 49.162335, -95.150834 49.162397, -95.150423 49.162404, -95.150009 49.162358, -95.149596 49.162312, -95.14918 49.162211, -95.148759 49.162003, -95.148341 49.161849, -95.147923 49.161695, -95.147505 49.161541, -95.147086 49.161387, -95.146668 49.161233, -95.14625 49.161078, -95.145832 49.160924, -95.145411 49.160716, -95.144993 49.160562, -95.144575 49.160408, -95.144157 49.160254, -95.143739 49.1601, -95.14332 49.159945, -95.142902 49.159791, -95.142484 49.159637, -95.142068 49.159537, -95.141655 49.15949, -95.141244 49.159498, -95.140832 49.159506, -95.140423 49.159567, -95.140012 49.159575, -95.139601 49.159583, -95.13919 49.15959),\nLINESTRING (-95.200201 49.100687, -95.200295 49.100955, -95.200471 49.101221, -95.200565 49.101489, -95.200659 49.101757, -95.200754 49.102026, -95.200848 49.102294, -95.200942 49.102562, -95.201036 49.10283, -95.20113 49.103098, -95.201224 49.103366, -95.2014 49.103633, -95.201494 49.103901, -95.201588 49.104169, -95.201682 49.104437, -95.201859 49.104703, -95.201953 49.104971, -95.202047 49.10524, -95.202141 49.105508, -95.202317 49.105774, -95.202411 49.106042, -95.202505 49.10631, -95.202599 49.106578, -95.202693 49.106847, -95.202788 49.107115, -95.202882 49.107383, -95.202976 49.107651, -95.20307 49.107919, -95.203246 49.108186, -95.20334 49.108454, -95.203434 49.108722, -95.203528 49.10899, -95.203705 49.109256, -95.203799 49.109524, -95.203893 49.109793, -95.203987 49.110061, -95.204081 49.110329, -95.204175 49.110597, -95.204269 49.110865, -95.204363 49.111133, -95.204457 49.111401, -95.204634 49.111668, -95.204728 49.111936, -95.204822 49.112204, -95.204916 49.112472, -95.205092 49.112738, -95.205186 49.113006, -95.20528 49.113275, -95.205375 49.113543, -95.205469 49.113811, -95.205563 49.114079, -95.205657 49.114347, -95.205751 49.114615, -95.205845 49.114883, -95.206021 49.11515, -95.206116 49.115418, -95.20621 49.115686, -95.206304 49.115954, -95.20648 49.11622, -95.206574 49.116489, -95.206668 49.116757, -95.206763 49.117025, -95.206857 49.117293, -95.206951 49.117561, -95.207045 49.117829, -95.207139 49.118097, -95.207233 49.118365, -95.207327 49.118633, -95.207339 49.118903, -95.207351 49.119173, -95.207363 49.119442, -95.207375 49.119712, -95.207387 49.119982, -95.207399 49.120251, -95.207329 49.120522, -95.207259 49.120794, -95.207189 49.121065, -95.207036 49.121338, -95.206884 49.121611, -95.206814 49.121882, -95.206661 49.122155, -95.206509 49.122427, -95.206439 49.122699, -95.206286 49.122971, -95.206134 49.123244, -95.206064 49.123515, -95.205911 49.123788, -95.205759 49.124061, -95.205689 49.124332, -95.205619 49.124604, -95.205466 49.124876, -95.205396 49.125148, -95.205326 49.125419, -95.205173 49.125692, -95.205021 49.125964, -95.204951 49.126236, -95.204798 49.126508, -95.204646 49.126781, -95.204576 49.127053, -95.204423 49.127325, -95.204271 49.127598, -95.204201 49.127869, -95.204048 49.128142, -95.203896 49.128415, -95.203826 49.128686, -95.203673 49.128959, -95.203521 49.129232, -95.203451 49.129503, -95.203298 49.129776, -95.203146 49.130049, -95.203076 49.13032, -95.202923 49.130593, -95.202771 49.130865, -95.202701 49.131137, -95.202548 49.131409, -95.202396 49.131682, -95.202325 49.131954, -95.202173 49.132226, -95.202021 49.132499, -95.20195 49.13277, -95.201798 49.133043, -95.201645 49.133316, -95.201575 49.133587, -95.201423 49.13386, -95.20127 49.134133, -95.201118 49.134406, -95.200965 49.134678, -95.200813 49.134951, -95.200661 49.135224, -95.200508 49.135497, -95.200438 49.135768, -95.200285 49.136041, -95.200133 49.136314, -95.200063 49.136585, -95.19991 49.136858, -95.199758 49.13713, -95.199688 49.137402, -95.199535 49.137674, -95.199383 49.137947, -95.199312 49.138218, -95.19916 49.138491, -95.199007 49.138764, -95.198937 49.139035, -95.198785 49.139308, -95.198632 49.139581, -95.198562 49.139852, -95.19841 49.140125, -95.198257 49.140398, -95.198187 49.140669, -95.198034 49.140942, -95.197882 49.141214, -95.197812 49.141486, -95.197659 49.141759, -95.197507 49.142031, -95.197436 49.142303, -95.197284 49.142575, -95.197131 49.142848, -95.197061 49.143119, -95.196909 49.143392, -95.196756 49.143665, -95.196604 49.143938, -95.196451 49.144211, -95.196216 49.144485, -95.196064 49.144758, -95.195829 49.145032, -95.195595 49.145306, -95.19536 49.145581, -95.195125 49.145855, -95.194808 49.146131, -95.194573 49.146405, -95.194339 49.14668, -95.194022 49.146956, -95.193705 49.147232, -95.19347 49.147506, -95.193153 49.147782, -95.192836 49.148058, -95.192519 49.148334, -95.192202 49.14861, -95.191885 49.148885, -95.191568 49.149161, -95.191251 49.149437, -95.190934 49.149713, -95.190617 49.149989, -95.190218 49.150267, -95.189819 49.150544, -95.18942 49.150821, -95.189021 49.151099, -95.188621 49.151376, -95.188222 49.151654, -95.187823 49.151931, -95.187424 49.152209, -95.187024 49.152486, -95.186625 49.152764, -95.186226 49.153041, -95.185827 49.153319, -95.185425 49.153542, -95.185023 49.153766, -95.184622 49.153989, -95.18422 49.154213, -95.183819 49.154436, -95.183419 49.154714, -95.183018 49.154937, -95.182616 49.155161, -95.182214 49.155384, -95.18181 49.155554, -95.181406 49.155723, -95.181002 49.155893, -95.180598 49.156062, -95.180194 49.156232, -95.17979 49.156402, -95.179386 49.156571, -95.178982 49.156741, -95.178578 49.15691, -95.178171 49.157026, -95.177767 49.157195, -95.177361 49.157311, -95.176955 49.157427, -95.176546 49.157488, -95.176139 49.157604, -95.175731 49.157666, -95.175322 49.157727, -95.174913 49.157789, -95.174506 49.157905, -95.174098 49.157966, -95.173689 49.158028, -95.173282 49.158144, -95.172876 49.158259, -95.172467 49.158321, -95.172061 49.158436, -95.171654 49.158552, -95.171245 49.158614, -95.170839 49.158729, -95.170433 49.158845, -95.170024 49.158907, -95.169617 49.159022, -95.169211 49.159138, -95.168802 49.159199, -95.168396 49.159315, -95.167989 49.159431, -95.16758 49.159492, -95.167174 49.159608, -95.166767 49.159723, -95.166358 49.159785, -95.165952 49.159901, -95.165546 49.160016, -95.165137 49.160078, -95.16473 49.160194, -95.164324 49.160309, -95.163915 49.160371, -95.163506 49.160432, -95.1631 49.160548, -95.162691 49.16061, -95.162282 49.160671, -95.161873 49.160733, -95.161464 49.160795, -95.161055 49.160856, -95.160647 49.160918, -95.160238 49.160979, -95.159829 49.161041, -95.15942 49.161103, -95.159011 49.161164, -95.158602 49.161226, -95.158193 49.161288, -95.157785 49.161349, -95.157376 49.161411, -95.156967 49.161472, -95.156558 49.161534, -95.156149 49.161596, -95.15574 49.161657, -95.155331 49.161719, -95.154923 49.16178, -95.154514 49.161842, -95.154105 49.161904, -95.153696 49.161965, -95.153287 49.162027, -95.152878 49.162089, -95.152469 49.16215, -95.152061 49.162212, -95.151652 49.162273, -95.151243 49.162335, -95.150834 49.162397, -95.150423 49.162404, -95.150009 49.162358, -95.149596 49.162312, -95.14918 49.162211, -95.148759 49.162003, -95.148341 49.161849, -95.147923 49.161695, -95.147505 49.161541, -95.147086 49.161387, -95.146668 49.161233, -95.14625 49.161078, -95.145832 49.160924, -95.145411 49.160716, -95.144993 49.160562, -95.144575 49.160408, -95.144157 49.160254, -95.143739 49.1601, -95.14332 49.159945, -95.142902 49.159791, -95.142484 49.159637, -95.142068 49.159537, -95.141655 49.15949, -95.141244 49.159498, -95.140832 49.159506, -95.140423 49.159567, -95.140012 49.159575, -95.139601 49.159583, -95.13919 49.15959),\n","geojson":{}}}

API requests fail with internal server error

Testing the API through AWS console results in an internal server error:

/v1/timeseries - GET method test results
Request
/v1/timeseries?feature=Reach&reach_id=71224100223&start_time=2022-08-04T00:00:00+00:00&end_time=2022-08-23T00:00:00+00:00&output=csv&fields=reach_id,time_str,wse

Latency
4173

Status
500

Response body

{"message": "Internal server error"}

Response headers

{
  "x-amzn-ErrorType": "InternalServerErrorException"
}

Log

Execution log for request e542f871-6bfe-49aa-a4ca-3c390284751b
Tue Nov 21 19:43:56 UTC 2023 : Starting execution for request: e542f871-6bfe-49aa-a4ca-3c390284751b
Tue Nov 21 19:43:56 UTC 2023 : HTTP Method: GET, Resource Path: /v1/timeseries
Tue Nov 21 19:43:56 UTC 2023 : Method request path: {}
Tue Nov 21 19:43:56 UTC 2023 : Method request query string: {output=csv, start_time=2022-08-04T00:00:00+00:00, reach_id=71224100223, feature=Reach, end_time=2022-08-23T00:00:00+00:00, fields=reach_id,time_str,wse}
Tue Nov 21 19:43:56 UTC 2023 : Method request headers: {}
Tue Nov 21 19:43:56 UTC 2023 : Method request body before transformations: 
Tue Nov 21 19:43:56 UTC 2023 : Endpoint request URI: https://lambda.us-west-2.amazonaws.com/2015-03-31/functions/arn:aws:lambda:us-west-2:1234:function:svc-hydrocron-sit-timeseries-lambda/invocations
Tue Nov 21 19:43:56 UTC 2023 : Endpoint request headers: {X-Amz-Date=20231121T194356Z, [TRUNCATED]
Tue Nov 21 19:43:56 UTC 2023 : Endpoint request body after transformations: {
  "body": {
    "exact":"",
    "region": "",
    "page_number": "" ,
    "page_size": "" ,
    "polygon_format": ""
  }
}
Tue Nov 21 19:43:56 UTC 2023 : Sending request to https://lambda.us-west-2.amazonaws.com/2015-03-31/functions/arn:aws:lambda:us-west-2:206226843404:function:svc-hydrocron-sit-timeseries-lambda/invocations
Tue Nov 21 19:44:00 UTC 2023 : Received response. Status: 200, Integration latency: 4168 ms
Tue Nov 21 19:44:00 UTC 2023 : Endpoint response headers: {Date=Tue, 21 Nov 2023 19:44:00 GMT, Content-Type=application/json, Content-Length=258, Connection=keep-alive, x-amzn-RequestId=8c03babd-e4a0-4dfd-9605-b4857ce465cf, X-Amz-Function-Error=Unhandled, x-amzn-Remapped-Content-Length=0, X-Amz-Executed-Version=$LATEST, X-Amzn-Trace-Id=root=1-655d087c-b96135890001f1bc20946ebf;sampled=0;lineage=125b713b:0}
Tue Nov 21 19:44:00 UTC 2023 : Endpoint response body before transformations: {"errorMessage": "'feature'", "errorType": "KeyError", "requestId": "8c03babd-e4a0-4dfd-9605-b4857ce465cf", "stackTrace": ["  File \"/var/task/hydrocron/api/controllers/timeseries.py\", line 186, in lambda_handler\n    feature = event['body']['feature']\n"]}
Tue Nov 21 19:44:00 UTC 2023 : Execution failed due to configuration error: Output mapping refers to an invalid method response: 500
Tue Nov 21 19:44:00 UTC 2023 : Method completed with status: 500

Update database load script to scramble data values on load

Options:

Create a shapefile that contains multiple timesteps for one or more river reaches/nodes
Scramble real data values from 1.1 (few repeat cycles) 1.0 (has more repeat cycles)

Decision:
Scramble (constant/random multiplication factor of the data values) the data from the SWOT_L2_HR_RiverSP_1.0 collection

Expose configuration option to turn the "scrambling" on/off.
Update any tests necessary to provide new parameter to load script

Add option for ingesting specific granule

Add option that allows a specific granule to be ingested.

Add SWORD version from shp.xml to DB entries

Stakeholder requested returning the SWORD version from the SWOT shapefiles as a field with the data attributes. This info is in a separate xml file in the shp zip so requires a bit of special handling.

Request format does not match spec

Spec requires feature_id:

hydrocron/terraform/api-specification-templates/hydrocron_aws_api.yml

Lines 26 to 28 in 35f0b11

 - name: feature_id 

 in: query 

 description: ID of the feature to retrieve in format CBBTTTSNNNNNN (i.e. 74297700000000)

API Looks for reach_id:

hydrocron/hydrocron/api/controllers/timeseries.py

Lines 180 to 185 in 35f0b11

 feature = event['body']['feature'] 

 feature_id = event['body']['reach_id'] 

 start_time = event['body']['start_time'] 

 end_time = event['body']['end_time'] 

 output = event['body']['output'] 

 fields = event['body']['fields']

API should be fixed to get the feature_id parameter.

Specification does not match actual returned content

The specification lists text/csv as return content

hydrocron/terraform/api-specification-templates/hydrocron_aws_api.yml

Lines 77 to 84 in 8a05c3c

 "200": 

 description: OK 

 content: 

 text/csv: 

 schema: 

 type: array 

 items: 

 type: string

Actual returned value is a json document:

{"status":"200 OK","time":"10 ms.","hits":10,"search on":{"parameter":"identifier","exact":"exact","page_number":0,"page_size":20},"results":"reach_id,time_str,wse\n77480300193,2023-10-24T00:09:09Z,5672.0592,\n"}

Need to bring specification in line with what the API is returning

Create lambda function to execute database loading

In order to run the database load script in AWS environment, it should be wrapped in a lambda instead of a command line tool.

Create Tutorial Documentation

Create a step-by-step guide for executing the API that can be included in the cookbook.

Fix Sonar code coverage calculation

Sonar is not correctly detecting the code coverage and always reports 0% covered.

Fix the scan so we get accurate code coverage metrics.

User guide for how to run database load script manually

Create documentation for operations specifying how to run the database load script manually.

Setup path for hydrocron under SOTO

Enable path based routing to add new endpoint

soto.podaac.sit.earthdatacloud.nasa.gov/hydrocron/v1/timeseries

SIT: https://bugs.earthdata.nasa.gov/servicedesk/customer/portal/7/NASD-3911
UAT: https://bugs.earthdata.nasa.gov/servicedesk/customer/portal/7/NASD-3952

Obscure data sometimes fails when 1 is chosen multiplier in randomization

1 should be excluded from the range of numbers to choose random multiplier

Create UMM-S record for hydrocron

Hydrocron is a new service being offered and should have a UMM-S record created describing what the service is.

AC:

UMM-S JSON file created and checked in to repo
Github workflow step added which updates UMM-S record as part of CICD

Setup performance benchmark test suite

Emulate users querying the API from public internet and characterize performance of the API.

Track status of granule ingest

Need to be able to determine which granules have been loaded into hydrocron and ensure that every granule ingested into CMR is also loaded in the database.

Setup github workflow to execute benchmark

Setup github workflow capable of running the benchmark suite

Merge API and DB into single repository

Merge database and api codebase and terraform into a single deployable hydrocron package

Encapsulate dynamodb under single shared module

Right now the dynamodb connection used by the API is setup here:

hydrocron/hydrocron/api/hydrocron.py

Line 67 in cb4bbe3

def construct_repository(self):

And the connection used for loading data is setup here:

hydrocron/hydrocron/db/load_data.py

Line 120 in cb4bbe3

def setup_dynamoconnection():

Both the API and DB Loading lambdas should share the same mechanism for creating the dynamo db connection

How do we indicate which collection version the data belongs to?

Users need to know which collection version the data is coming from

Implement benchmark tests

Fill out the tests to actually execute live requests against a target environment

Update log messaging format

EDC Metrics will provide the messaging format we should use. Need to update code to use correct messaging format

Support all shp attributes in fields parameter of query

Feature type = reach queries only return reachid, time, wse, when CSV format chosen and reachid, geometry, time, wse, slope when GEOJSON format chosen. Feature type = node is not supported at all in GeoJSON format.

All fields from both reach and node shapefiles should be supported in any combination, and if no fields parameter is specified, all attributes should be returned.

Cleanup api code

Remove commented out code.
Remove unused parameters
Clean up todo comments

Create documentation

Create Docker container for lambda functions

Lambda functions should use a lambda container runtime docker container.

Enable notification of swot shapefile ingest to tva

I&A update cumulus configurations to send CNM-R messages to TVA SNS Topic

I&A Requirements:

ARN for SNS topic that CNM-R messages should be sent to

TVA Services requirements:

Allow I&A permission to publish to SNS
Use s3 credentials direct access to download file

TODO:

Specific collection shortname/concept IDs

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

	- name: feature_id
	in: query
	description: ID of the feature to retrieve in format CBBTTTSNNNNNN (i.e. 74297700000000)

	feature = event['body']['feature']
	feature_id = event['body']['reach_id']
	start_time = event['body']['start_time']
	end_time = event['body']['end_time']
	output = event['body']['output']
	fields = event['body']['fields']

	"200":
	description: OK
	content:
	text/csv:
	schema:
	type: array
	items:
	type: string