Source URIs are taking longer than expected to return result. Up connection time out to 60sec.

If the attributes declared in the crawler source table don't exist in the source data, the crawler goes ahead without error. We should verify that the declared attributes exist before continuing with the crawl.

We have a mirror in place for this repository which will allow us to run build and deploy pipelines internally. The Jenkins files should be removed and replace with a .gitlab-ci.yml with an include: section that references the internal pipelines. As part of this, we can also remove the .travis.yml file.

some stuff, some more reasonable than others

• social login options. maybe just github + google
• nested RBAC w/ organization and user levels similar to github orgs
  workspace listing each dataset
• wizard to add dataset link, configure which columns correspond to required fields, perhaps with preview of header + first two lines of target dataset
• option to upload dataset to some default space (subject to reasonable storage limits)
• "refresh" buttons to initiate recrawl of any datasets that don't need to change configuration
• Stats
  • total calls involving each dataset over time (line-graph)
  • Call counts of each dataset under user's management as originating point with each other NLDI dataset as navigation target
  • Call counts of each NLDI dataset as originating point with each dataset under user's management as navigation target

reach goal: option to generate URIs as geoconnex IDs using a namespace/stem text entry + column select for id, w/ prompt to go finalize at geoconnex github/ TBD similar system.

A GitHub Action should be created to handle testing a pull request. This will help us resolve dependabot updates more quickly.

Not sure if this is the right place to ask for this feature, but when doing up/downstream queries it is often necessary if there is a dam in place.

In the case that a feature in a set (such as a stream gage) is far from a stream and does not have a river address, we should be able to "fall" back to catchment indexing.

See internetofwater/ref_gages#22 for some example gages.

To support TMDLs and other linear events, a crawler should be able pull in a linear feature that has a linked comid.

Note that this is a many to many relationship between lines and catchments.

Per https://github.com/ACWI-SSWD/nldi-crawler/blob/master/src/main/java/gov/usgs/owi/nldi/service/Ingestor.java#L160 the crawler isn't looking at an "id" that's not in the properties.

Standard geoJSON supports an id at the level of the actual feature.

e.g. https://locations.newmexicowaterdata.org/collections/Things/items/4201?f=json

Seems like we could add the "id" to the properties if there isn't an "id" in the properties already here. https://github.com/ACWI-SSWD/nldi-crawler/blob/master/src/main/java/gov/usgs/owi/nldi/service/Ingestor.java#L170

There are several tests that need a _temp table to exist and preload it with data. The @DatabaseSetup annotation does not allow for table creation, only data injection. An approach needs to be found that allows for temporarily creating these tables in the CI database before the tests are run.

My initial thought would be to have a @BeforeClass (or similar) function that runs a SQL command to create the necessary table(s). I have tried a few variations of this, but did not get a successful run. Further research needs to be done to determine whether we could use an available annotation, or potentially grab the jdbc connection to run the command.

A workaround for this issue was implemented in #141

Currently, the crawler keeps all records that are read in whether they get indexed or not. The crawler should operate exclusively where it only keeps data that indexes to a comid.

When a crawl finishes, no rows with NULL comids should remain in the NLDI database. This could be made configurable but default to drop un-indexed features.

Question is how to handle table names based on source_suffix value. If the suffix has a dash in the name, does that work with postgres table names?

prefer to raise an error rather than silently coerce the table name.

I want to write a dockerized version of the crawler in R so I can contribute crawler code myself. Having a pattern for both python and R would be really nice. I doubt it would be too heavy a lift but need to do a little research on how it would work out.

With the larger tables from wade and the census blocks, it's becoming clear that we need indexes on the feature tables.

The crawler should be adding a comid index to help improve join performance.

https://github.com/internetofwater/nldi-crawler/blob/master/src/main/resources/mybatis/mappers/ingest.xml#LL30C4-L30C4 seems to be where it is being handled during ingest.

It looks like it maybe adopts from this? https://github.com/internetofwater/nldi-db/blob/d372160066d343ddd671124049a6e2e95488c1b2/liquibase/changeLogs/nldi/nldi_data/indexes/featureId.yml#L4 ?

Maybe @EthanGrahn can pop in and give us a pointer on a quick way to add a comid index?

The NLDI database will have a new "shape" column that supports generic geometry points. The crawler should allow ingestion into either the current "location" column or the new "shape" column.

Moving conversation from this issue here.

I'm generally uncomfortable with the semantics of lakes as wide rivers have many (all?) of the same characteristics from a data model point of view and need to be treated on the same spectrum of data.

I'm confused, so are you saying lakes are treated as wide rivers now? I think that would actually be a fairly reasonable way to do that. Especially if the lake catchments were calculated and up-catchment datapoints were linked to the lake itself. Doesn't really matter if the system knows it is a lake or not, or just a really wide river.

Add type annotations and hints such that the codebase passes mypy 'strict' type checking.

The Crawler integration tests had an expectation of data and tables that were already in the CI database. The CI database no longer contains any data or temp tables. Integration tests need to be updated to create the data they expect so that they can use the latest CI Docker image.

A NLDI-ready geojson of sites from the WaDE 2.0 services is available here: https://www.hydroshare.org/resource/5f665b7b82d74476930712f7e423a0d2/

source_name: Water Data Exchange 2.0 Sites
source_suffix: wade
source_uri: https://www.hydroshare.org/resource/5f665b7b82d74476930712f7e423a0d2/data/contents/wade.geojson
feature_id: feature_id
feature_name: feature_name
feature_uri: feature_uri
feature_reach: NULL
feature_measure: NULL
ingest_type: point

If the team is not comfortable adding this to production, it would be ok in the near term to have it on beta/QA. It includes about 1M points.

We need to add including indexes to this section of SQL so that feature table indexes are preserved.

The crawler creates a temporary table during ingestion and is supposed to drop the table once complete. I've noticed that there are temp tables in our deployed database that did not get cleaned up. (different issue from internetofwater/nldi-db#36) There needs to be investigation on what causes the cleanup to fail.