mikejohnson51 / hydrodata Goto Github PK

View Code? Open in Web Editor NEW

39.0 10.0 5.0 244.8 MB

An R 📦 for finding and getting geospatial earth systems data

Home Page: https://mikejohnson51.github.io/HydroData/

License: MIT License

R 100.00%

hydrology noaa usgs weather dataset climate aoi r

hydrodata's Introduction

HydroData: Earth Systems Data in R

HydroData is designed to help (1) find, (2) get, (3) visualize, and (4) format disparate earth systems data through a core language (R); a common geospatial reference (ESPG:4629) ; and unifying vocabulary built around querying data by an area of interest (AOI). The package supports access to 20+ National/Global data sources.

All functions are designed to work with the AOI package and magrittr pipe operation %>% allowing successive HydroData calls to be directly chained to an area of interest:

Number	Dataset	Description	Format
1	GHCN	Global Historical Climate Network	Vector point
2	International Airports	OpenFlights Geo-database	Vector point
3	NID	National Inventory of Dams (USACE)	Vector point
4	NWIS	National Water Information System Site Index	Vector point
5	Snotel	The NRCS Snow Measurement Stations	Vector point
6	GagesII	Geospatial Attributes of Gages for Evaluating Streamflow	Vector point, line
7	NLDI	The Network Linked Data Index	Vector point, line, polygon
8	TIGER	2017 US Census Bureau Road Network	Vector line
9	NHDPlus	National Hydrography Dataset Waterbodies, River Network, Catchments	Vector line, polygon
10	EPA Basins	Environmental Protection Agency NWIS drainage basins	Vector polygon
11	SSURGO	National Cooperative Soil Survey Geospatial Soil Data Base	Vector polygon
12	WBD	The Watershed Boundary Dataset (all HUC levels)	Vector polygon
13	CDL	USDA Crop Data Layers (2008 - 2017)	Raster
14	NED	National Elevation Dataset (10 and 30 meter resolutions)	Raster
15	NLCD	National Land Cover Dataset (01,06,11) (Impervious, canopy, Land cover)	Raster

myData = getAOI(clip = list("UCSB", 10, 10)) %>% 
findNHD() %>% 
findWaterbodies %>% 
findNWIS %>% 
findNED()

str(myData,max.level = 2)

List of 5
 $ AOI        :Formal class 'SpatialPolygons' [package "sp"] with 4 slots
 $ nhd        :Formal class 'SpatialLinesDataFrame' [package "sp"] with 4 slots
 $ waterbodies:Formal class 'SpatialPolygonsDataFrame' [package "sp"] with 5 slots
 $ nwis       :Formal class 'SpatialPointsDataFrame' [package "sp"] with 5 slots
 $ NED        :Formal class 'RasterLayer' [package "raster"] with 12 slots
 - attr(*, "class")= chr "HydroData"

For now, most internal HydroData operations are based of the 'simple feature (sf)' library however objects are all returned as Spatial* sp objects. As the community around simple features grows the default output of 'HydroData' might change. Until then users can use the to_sf function to convert a HydroData object to simple features where appropriate.

HydroData offer in-package tools for generating interactive visualizations of HydroData Spatial* objects

myData.sf = myData %>% to_sf

str(myData.sf,max.level = 1)

List of 5
 $ NED        :Formal class 'RasterLayer' [package "raster"] with 12 slots
 $ AOI        :Classes ‘sf’ and 'data.frame':	1 obs. of  1 variable:
  ..- attr(*, "sf_column")= chr "geometry"
  ..- attr(*, "agr")= Factor w/ 3 levels "constant","aggregate",..: 
  .. ..- attr(*, "names")= chr(0) 
 $ nhd        :Classes ‘sf’ and 'data.frame':	100 obs. of  92 variables:
  ..- attr(*, "sf_column")= chr "geometry"
  ..- attr(*, "agr")= Factor w/ 3 levels "constant","aggregate",..: NA NA NA NA NA NA NA NA NA NA ...
  .. ..- attr(*, "names")= chr [1:91] "id" "ogc_fid" "comid" "fdate" ...
 $ waterbodies:Classes ‘sf’ and 'data.frame':	11 obs. of  24 variables:
  ..- attr(*, "sf_column")= chr "geometry"
  ..- attr(*, "agr")= Factor w/ 3 levels "constant","aggregate",..: NA NA NA NA NA NA NA NA NA NA ...
  .. ..- attr(*, "names")= chr [1:23] "id" "objectid" "comid" "fdate" ...
 $ nwis       :Classes ‘sf’ and 'data.frame':	3 obs. of  8 variables:
  ..- attr(*, "sf_column")= chr "geometry"
  ..- attr(*, "agr")= Factor w/ 3 levels "constant","aggregate",..: NA NA NA NA NA NA NA
  .. ..- attr(*, "names")= chr [1:7] "OBJECTID" "feature_id" "site_no" "site_name" ...

HydroData offer in-package tools for generating interactive visualizations of HydroData Spatial* objects

 myData %>% explore()

To download and get started with HydroData, install from GitHub using the devtools packages:

library(devtools)
install_github("mikejohnson51/HydroData")

Support:

HydroData is written by Mike Johnson, a graduate Student at the University of California, Santa Barbara in Keith C. Clarke's Lab, 2018 and is funded through the NOAA National Water Center (NWC) via the UCAR COMET Program (2017/18).

hydrodata's People

Contributors

Stargazers

Watchers

Forkers

keyao22 minasel jsta kkyong77 ntag1618

hydrodata's Issues

testthat framework setup

Just get a single test working within a basic testthat framework

Does HydroData actually use sf package?

sf is great. I use the heck out of it, but it's a little tricky with travis testing. Is it actually used in the package right now?

Can't seem to get it downloaded

Looks like a pretty sweet package! AOI works just fine but can't seem to get HydroData to successfully download/install. Any help would be much appreciated, thanks.

Session Info

devtools::session_info()
#> - Session info ---------------------------------------------------------------
#>  setting  value                       
#>  version  R version 4.0.2 (2020-06-22)
#>  os       Windows 10 x64              
#>  system   x86_64, mingw32             
#>  ui       RTerm                       
#>  language (EN)                        
#>  collate  English_United States.1252  
#>  ctype    English_United States.1252  
#>  tz       America/Denver              
#>  date     2020-10-05                  
#> 
#> - Packages -------------------------------------------------------------------
#>  package     * version date       lib source        
#>  assertthat    0.2.1   2019-03-21 [1] CRAN (R 4.0.2)
#>  backports     1.1.10  2020-09-15 [1] CRAN (R 4.0.2)
#>  callr         3.4.4   2020-09-07 [1] CRAN (R 4.0.2)
#>  cli           2.0.2   2020-02-28 [1] CRAN (R 4.0.2)
#>  crayon        1.3.4   2017-09-16 [1] CRAN (R 4.0.2)
#>  curl          4.3     2019-12-02 [1] CRAN (R 4.0.2)
#>  desc          1.2.0   2018-05-01 [1] CRAN (R 4.0.2)
#>  devtools    * 2.3.1   2020-07-21 [1] CRAN (R 4.0.2)
#>  digest        0.6.25  2020-02-23 [1] CRAN (R 4.0.2)
#>  ellipsis      0.3.1   2020-05-15 [1] CRAN (R 4.0.2)
#>  evaluate      0.14    2019-05-28 [1] CRAN (R 4.0.2)
#>  fansi         0.4.1   2020-01-08 [1] CRAN (R 4.0.2)
#>  fs            1.5.0   2020-07-31 [1] CRAN (R 4.0.2)
#>  glue          1.4.2   2020-08-27 [1] CRAN (R 4.0.2)
#>  highr         0.8     2019-03-20 [1] CRAN (R 4.0.2)
#>  htmltools     0.5.0   2020-06-16 [1] CRAN (R 4.0.2)
#>  knitr         1.29    2020-06-23 [1] CRAN (R 4.0.2)
#>  magrittr      1.5     2014-11-22 [1] CRAN (R 4.0.2)
#>  memoise       1.1.0   2017-04-21 [1] CRAN (R 4.0.2)
#>  pkgbuild      1.1.0   2020-07-13 [1] CRAN (R 4.0.2)
#>  pkgload       1.1.0   2020-05-29 [1] CRAN (R 4.0.2)
#>  prettyunits   1.1.1   2020-01-24 [1] CRAN (R 4.0.2)
#>  processx      3.4.4   2020-09-03 [1] CRAN (R 4.0.2)
#>  ps            1.3.4   2020-08-11 [1] CRAN (R 4.0.2)
#>  R6            2.4.1   2019-11-12 [1] CRAN (R 4.0.2)
#>  remotes       2.2.0   2020-07-21 [1] CRAN (R 4.0.2)
#>  rlang         0.4.7   2020-07-09 [1] CRAN (R 4.0.2)
#>  rmarkdown     2.3     2020-06-18 [1] CRAN (R 4.0.2)
#>  rprojroot     1.3-2   2018-01-03 [1] CRAN (R 4.0.2)
#>  sessioninfo   1.1.1   2018-11-05 [1] CRAN (R 4.0.2)
#>  stringi       1.5.3   2020-09-09 [1] CRAN (R 4.0.2)
#>  stringr       1.4.0   2019-02-10 [1] CRAN (R 4.0.2)
#>  testthat      2.3.2   2020-03-02 [1] CRAN (R 4.0.2)
#>  usethis     * 1.6.1   2020-04-29 [1] CRAN (R 4.0.2)
#>  withr         2.3.0   2020-09-22 [1] CRAN (R 4.0.2)
#>  xfun          0.18    2020-09-29 [1] CRAN (R 4.0.2)
#>  yaml          2.2.1   2020-02-01 [1] CRAN (R 4.0.2)
#> 
#> [1] C:/R/R-4.0.2/library

Reprex

library(devtools)
#> Loading required package: usethis
install_github("mikejohnson51/HydroData")
#> Downloading GitHub repo mikejohnson51/HydroData@HEAD
#> 
#>          checking for file 'C:\Users\joshualerickson\AppData\Local\Temp\RtmpMlfQ1M\remotes4c6478702b50\mikejohnson51-HydroData-6b7f086/DESCRIPTION' ...  v  checking for file 'C:\Users\joshualerickson\AppData\Local\Temp\RtmpMlfQ1M\remotes4c6478702b50\mikejohnson51-HydroData-6b7f086/DESCRIPTION' (450ms)
#>       -  preparing 'HydroData': (1.4s)
#>    checking DESCRIPTION meta-information ...     checking DESCRIPTION meta-information ...   v  checking DESCRIPTION meta-information
#>       -  checking for LF line-endings in source and make files and shell scripts
#>       -  checking for empty or unneeded directories
#>       -  building 'HydroData_0.0.7.tar.gz'
#>      
#> 
#> Error: Failed to install 'HydroData' from GitHub:
#>   (converted from warning) installation of package 'C:/Users/JOSHUA~1/AppData/Local/Temp/RtmpMlfQ1M/file4c6420e71c75/HydroData_0.0.7.tar.gz' had non-zero exit status

other info

* installing *source* package 'HydroData' ...
** using staged installation
** R
** data
*** moving datasets to lazyload DB
** byte-compile and prepare package for lazy loading
Error: package or namespace load failed for 'AOI':
 .onLoad failed in loadNamespace() for 'sf', details:
  call: CPL_gdal_init()
  error: (converted from warning) GDAL Error 1: Can't load requested DLL: C:\Program Files (x86)\GDAL\gdalplugins\gdal_BAG.dll
193: %1 is not a valid Win32 application.

Error: package 'AOI' could not be loaded
Execution halted
ERROR: lazy loading failed for package 'HydroData'
* removing 'C:/R/R-4.0.2/library/HydroData'
Error: Failed to install 'HydroData' from GitHub:
  (converted from warning) installation of package ‘C:/Users/JOSHUA~1/AppData/Local/Temp/Rtmp4AbiEn/file75487c4e401e/HydroData_0.0.7.tar.gz’ had non-zero exit status

Add improved naming convention to explore and inspect using define_clip_unit/name_clip_unit

set up pkgdown site

New MET data sources

WeatherUnderground has changed there website from static to dynamic

New options are the IOWA mesonet and the EU site. This require airports to be called by their SYNOP rather then ICAO id.

Implement starter vignette and link to rawgit view of it in readme

probably best to use "dev" rawgit since the "production" rawgit doesn't update through time very well.

travis setup

@dblodgett-usgs will get this working over on his repo then we'll transfer it here.

move towards httr, progress, crayon outputs

see example: from nwm package

add return_what call to all applicable functions

able to remove baseman, boundary, and id redundancies

Add save as csv to save function

If data is tabular then should be exported to CSV

add HydroDate class to each function

A HydroData object should be a list with:

site
source
data

coveralls setup

@dblodgett-usgs will get this setup over in his repo then we'll transfer here.

Find WBD fails on large queries

getAOI(state = c("RI")) %>% findWBD()

works

getAOI(state = c("MI")) %>% findWBD()

fails with error:
Cannot open data source https://hydro.nationalmap.gov/arcgis/rest/services ...
Error in CPL_read_ogr(dsn, layer, query, as.character(options), quiet, :
Open failed.
In addition: Warning message:
In CPL_read_ogr(dsn, layer, query, as.character(options), quiet, :
GDAL Error 4: Failed to read GeoJSON data

library(sf)
library(HydroData)
bb <- st_as_sfc(st_bbox(c(xmin = -97.36394, xmax = -89.70397,
                ymax = 46.35289, ymin = 40.31668), crs = st_crs(4326)))

h4s <- findWBD(getAOI(bb), level = 4, crop = FALSE)
all(sapply(st_overlaps(st_as_sf(h4s$huc4)), length) == 0)