Comments (10)
mikejohnson51
commented on May 29, 2024
Hi @eli-asarian,
Thanks for the note! I would be game to add these datasets. Unfortunately we need them exposed as OpenDAP services rather than direct data downloads. I have done a little "googling" and not found an endpoint. Are you aware of one?
Mike
from climater.
eli-asarian
commented on May 29, 2024
Ah, I thought that might be the case. Unfortunately no, I'm not aware of any OpenDAP service for the NEX-GDM. Ever since that paper came out a few years ago, I've been periodically checking to find out where to download the data (was not initially available), finally found it today! So I think that's the only location that the data are available.
from climater.
mikejohnson51
commented on May 29, 2024
Ah bummer. One bright note is that I am slowly working on generalizing the features of climateR here: https://github.com/mikejohnson51/opendap.catalog
While it won't help with the downloading, it will give you climateR syntax to local files. If you want to transfer this issue over there I would be happy to make it a "main" test case?
Mike
from climater.
eli-asarian
commented on May 29, 2024
Mike, I'm not really familiar enough with GitHub, OpenDAP, etc. to be sure I understand what you mean, so apologies for any mis-understandings. I don't necessarily need NEX-GDM for the project I'm working on (statistical modeling of water temperatures in rivers in northern California and Oregon), so I'd rather not put more than several hours of work into it. PRISM, GridMET, and Daymet will be fine for my project, though I'm always looking for interesting new datasets! But I think you're saying that one of us (not sure if it's you or me) would put a comment on the GitHub page for opendap.catalog, and that you'd then follow up by testing your tool on some NEX-GDM data and posting that code as an example? And then I would follow-up by testing your example and reporting back on whether it worked for me? That sounds fine to me.
from climater.
mikejohnson51
commented on May 29, 2024
Hi @eli-asarian, This is good enough to remind me :) So check out where we are heading:
You'll notice "URL" is (poorly named now) but one of the files downloaded from the site you provided. The idea is with a more general OpenDap implementation, and the opinionated climateR inputs, you can get the same results from local resources the package doesn't know about. This is getting close to being ready for use.
Mike
from climater.
mikejohnson51
commented on May 29, 2024
Following up w/ progress:
library(terra)
#> terra 1.5.12
library(opendap.catalog)
url <- '/Users/mjohnson/Downloads/NEXGDM_srad_2020_v100.nc'
utils:::format.object_size(file.size(url), "auto")
#> [1] "3.7 Gb"
system.time({
dap = dap_crop(URL = url,
AOI = AOI::aoi_get(state = "FL"),
startDate = "2020-01-01",
endDate = "2020-01-05")
out = dap_get(dap)
})
#> Warning in getGeoDatum(gm): Didn't find a longitude of prime meridian for datum,
#> assuming 0.
#> Warning in getGeoDatum(gm): Didn't find a semi major axis for datum, assuming
#> WGS84 6378137.0 meters
#> Warning in getGeoDatum(gm): Didn't find an inverse flattening value, assuming
#> WGS84 298.257223563
#> Warning in variable_meta(catolog, verbose = FALSE): raw must include variable
#> column
#> Warning in variable_meta(catolog, verbose = FALSE): trying varname. Chance of
#> failure...
#> user system elapsed
#> 1.093 0.134 1.246
opendap.catalog:::print.dap(dap)
#> vars: > srad [NA]
#> X: 807 (x)
#> Y: 693 (y)
#> T: 4 (time - 1 days)
#> values: 2,237,004 (vars*X*Y*T)
terra::plot(out$srad)
Created on 2022-01-24 by the reprex package (v2.0.1)
from climater.
eli-asarian
commented on May 29, 2024
Thanks Mike! What functions would I then use to extract a time series of srad for a site (or even better- a collection of sites)?
from climater.
mikejohnson51
commented on May 29, 2024
I dont have that built into a function yet but it could be done like this!
library(opendap.catalog)
library(terra)
#> terra 1.5.12
# Find 100 random points in Florida
rando_sites <- AOI::aoi_get(state = "FL", county = "all") |>
sf::st_sample(100) |>
sf::st_as_sf()
#Give them an ID
rando_sites$ID = 1:100
plot(rando_sites['ID'], pch = 16)
# Use Sites as AOI
nexgdm = dap_crop(URL = '/Users/mjohnson/Downloads/NEXGDM_srad_2020_v100.nc',
AOI = rando_sites,
startDate = "2020-01-01", endDate = "2020-01-31") |>
dap_get()
# Extract
long = extract(nexgdm$srad, project(vect(rando_sites), terra::crs(nexgdm$srad)))
head(long)
#> ID 2020-01-02 2020-01-03 2020-01-04 2020-01-05 2020-01-06 2020-01-07
#> 1 1 12.51465 11.433594 9.731445 12.146484 14.23926 13.66699
#> 2 2 13.30273 12.774414 11.503906 10.379883 15.15527 14.55859
#> 3 3 10.70605 9.842773 8.231445 6.630859 13.41309 13.01758
#> 4 4 11.27930 10.337891 10.535156 5.708008 13.28125 13.23828
#> 5 5 10.58203 9.267578 5.909180 8.864258 13.23730 12.76953
#> 6 6 12.71387 12.476562 10.421875 9.011719 14.51367 14.34863
#> 2020-01-08 2020-01-09 2020-01-10 2020-01-11 2020-01-12 2020-01-13 2020-01-14
#> 1 13.57812 14.08008 11.79492 6.241211 10.550781 10.946289 10.817383
#> 2 14.22363 13.82910 11.70703 7.019531 9.934570 11.211914 10.026367
#> 3 12.54688 13.45898 11.72168 7.107422 7.457031 8.614258 8.974609
#> 4 12.65723 13.50391 11.82324 7.129883 9.266602 8.270508 10.487305
#> 5 11.68945 13.49414 10.60840 7.038086 3.723633 6.588867 6.015625
#> 6 13.79199 13.73047 11.89453 6.439453 10.306641 10.143555 11.301758
#> 2020-01-15 2020-01-16 2020-01-17 2020-01-18 2020-01-19 2020-01-20 2020-01-21
#> 1 10.229492 12.562500 12.678711 9.460938 10.999023 12.185547 10.93945
#> 2 9.205078 14.024414 12.739258 8.768555 11.688477 13.194336 13.54688
#> 3 7.958984 8.573242 8.313477 11.371094 8.815430 7.308594 13.86523
#> 4 7.258789 10.922852 10.262695 9.558594 10.215820 6.975586 13.02051
#> 5 8.473633 7.531250 7.897461 11.611328 8.452148 6.739258 14.07520
#> 6 9.989258 13.538086 12.821289 9.030273 11.199219 12.825195 13.86426
#> 2020-01-22 2020-01-23 2020-01-24 2020-01-25 2020-01-26 2020-01-27 2020-01-28
#> 1 14.13086 12.390625 9.960938 9.906250 12.58008 13.674805 7.528320
#> 2 15.84082 11.463867 12.430664 8.785156 14.54102 15.129883 6.277344
#> 3 13.96289 12.716797 6.490234 6.985352 12.81055 8.961914 5.953125
#> 4 13.26953 8.676758 6.372070 8.671875 13.51172 10.281250 6.125977
#> 5 14.26758 10.905273 5.687500 8.631836 12.25586 5.284180 5.945312
#> 6 15.27148 12.758789 11.615234 9.223633 14.43262 14.905273 6.788086
#> 2020-01-29 2020-01-30 2020-01-31
#> 1 13.22461 11.425781 13.710938
#> 2 13.64355 11.977539 11.243164
#> 3 13.11719 5.265625 11.782227
#> 4 13.74902 6.803711 9.189453
#> 5 14.53711 4.712891 11.766602
#> 6 13.72852 11.896484 13.925781
# Piviot (if desired)
wide = data.frame(date = as.Date(names(long)[-1]), setNames(data.frame(t(long[,-1])), paste0("site_", long[,1])))
head(wide[,1:5])
#> date site_1 site_2 site_3 site_4
#> 2020-01-02 2020-01-02 12.514648 13.30273 10.706055 11.279297
#> 2020-01-03 2020-01-03 11.433594 12.77441 9.842773 10.337891
#> 2020-01-04 2020-01-04 9.731445 11.50391 8.231445 10.535156
#> 2020-01-05 2020-01-05 12.146484 10.37988 6.630859 5.708008
#> 2020-01-06 2020-01-06 14.239258 15.15527 13.413086 13.281250
#> 2020-01-07 2020-01-07 13.666992 14.55859 13.017578 13.238281
plot(wide$date, wide$site_1, type = "l")
Created on 2022-02-02 by the reprex package (v2.0.1)
from climater.
eli-asarian
commented on May 29, 2024
Excellent, thanks so much for your help!
from climater.
mikejohnson51
commented on May 29, 2024
Hi @eli-asarian. This has all been merged into this package now. Hope it helps!
from climater.
Related Issues (20)
- Error in masking HOT 1
- getGLDAS error "Access denied" HOT 19
- We are missing an ensemble filter. HOT 1
- Add utility to animate outputs HOT 5
- devtools::check() error found in test-netrc.R HOT 1
- Daymet return data date range is off by 1 HOT 2
- Add Average Option to PRISM Air Temperature HOT 2
- Odd dates returned HOT 2
- Added "../.." duration support in the catalog.
- Best way to get prism data for a single point HOT 2
- Question on using the precip data HOT 10
- Question: Is there a way to get annual means from gridMET data using getGridMet?
- getGridMET returns url with class "glue" "character" HOT 4
- no more srad data from TerraClim? HOT 9
- getGridMET returns error when embedded within another function HOT 2
- Accept sfc again
- Trouble Downloading over large area with SpatVect AOI HOT 8
- Question and Help: On How to Plot Monthly Data into Yearly Data In Tempreture change Map using getTerra climate HOT 10
- Problem download MODIS data
- ClimateR freezes
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from climater.