ropensci / gsodr Goto Github PK

When specifying years if years are requested using get_GSOD() if data is not available for years requested, the function still attempts a download and returns an empty object.

Should stop when no files are found for years requested.

I can retrieve the station data as a data frame when I first open my script, but only once. After that, I get this error for any further efforts to retrieve station data, whether as a data frame or as a csv:

Is this an issue with the server?

Enable selection of stations within a radius of a given lon/lat point or use bounding box to select stations for download.

Hi Adam,

in order for update-tic.yml to work, you need to add a secret with "workflow" scopes named "TIC_UPDATE" to your repo.

This can be easily done via tic::gha_add_secret(<secret>, "TIC_UPDATE").

Just letting you know since you are one of the few persons who were keen enough to try this new functionality :)

could you please make sure to git tag each CRAN releasehttps://github.com/ropensci/GSODR/tags and plop in associated NEWS items for that release - thanks! e.g., https://github.com/ropensci/taxize/releases/tag/v0.9.3

Dear Sparks,
Thanks for the GSODR package tutorial provided.
Using the Global Summary of the Day data from NOAA, I want to plot temperature (mean daily minimum temperatures, mean daily maximum temperatures), precipitation (total precipitation, average precipitation) for three Stations in Tanzania from 1957 to 2016 (station IDs: Kigoma, Zanzibar, and Mtwara).
Then repeat the process for the Tanzania as a country.
I read the example for Toowoomba, Queenland for 2010 provided on github. I wanted to start with your example and then adapt the codes to solve my problems. However, I have not been able to do this. I also tried to replicate the exact examples as presented for Toowoomba, Queenland for 2010 but could not still do it.
See my attempts below with the message I get. I will appreciate any assistance with how to plot the variables described above for the three stations in Tanzania and for the Country Tanzania from 1957 to 2016. I am not proficient in the use of R.

Thanks .

@PRoyal22 suggested to be able to merge CSV files for all years of a single station that are queried.

This is a desirable and easy to implement feature for a single station and include in the get_GSOD() function. To do this for all years queried for the complete GSOD set or agroclimatology might not be desirable simply due to the size of the final files that are generated for each year.

get_GSOD(years = 2010, country = "Australia", path = "~/Downloads") results in an empty CSV file.

mean wind speed (WDSP) is wrongly recalculated from knots to m/s.

@sckott,

I didn't edit the link to the webpage before moving the repository from my page to ropensci and now the link points back at the old location but I do not have the ability to edit it.

Global Summary Daily Weather Data in R http://adamhsparks.github.io/GSODR/

should be

Global Summary Daily Weather Data in R http://ropensci.github.io/GSODR/

The data are now available in CSV files by year/station over https!

https://www.ncei.noaa.gov/data/global-summary-of-the-day/access/

MIN description refers to MAX.

Precipitation is calculated incorrectly as cm not mm.

This commit fixes that.
8d09f2257f982de4fa6ea492248124acbe0a4fc7

Thanks to Gwenael Giboire (@GwenGitHub) for reporting this bug.

Prepare for release:

• Check current CRAN check results
• Polish NEWS
• urlchecker::url_check()
• devtools::check(remote = TRUE, manual = TRUE)
• devtools::check_win_devel()
• rhub::check_for_cran()
• revdepcheck::revdep_check(num_workers = 4)
• Update cran-comments.md

Submit to CRAN:

• usethis::use_version('minor')
• devtools::submit_cran()
• Approve email

Wait for CRAN...

• Accepted 🎉
• usethis::use_github_release()
• usethis::use_dev_version()
• Tweet

With the newly rewritten functionality test coverage has taken a hit. Need to write better unit tests.

Similar to selecting years, add ability to select more than one station to download at one time.

@karthik the new badge for the getCRUdata package works, but for GSODR it's not appearing.

I can't find anything in the code that would prevent this from working.

Any thoughts?

I tried to download data from 2 stations with this piece of code

install.packages("GSODR")
library(GSODR)
library(dplyr)

load(system.file("extdata", "country_list.rda", package = "GSODR"))
load(system.file("extdata", "isd_history.rda", package = "GSODR"))

a <- get_GSOD(years = 2010, station = "489300-99999")    
b <- get_GSOD(years = 2010, station = "489260-99999")

I don't understand why b contains data from a:

I used package version 1.3.1 and R version 3.5.0.

Many thanks for your help!

Thanks for doing this
Couple of points

1. There does not appear to be any difference in time taken to return a list of stations frrom a distance of 10 or 100 km . In my test 4 are returned in case 1 and 107 in case 2. both take around 10 secs. Is this just the way it is or could the former be speeded up

2. The r code implies that you could return a distance (miles=FALSE). Can you set an option so that distance could also be returned as station id (presumably as a data.frame)so that the nearest or nearest 5,say, could be identified for further processing

thank

• I am trying to download data for a specific year-station. For example:

Lond_2005 <- get_GSOD(years = 2005, station = "837660-99999")

It is returning only 2010 irrespectively of the chosen year.

$ YEAR <chr> "2010", "2010", "2010", "2010", "2010", "2010",...

• Is there a way to download data from multiple/interval of years? something like c(2010:2014)

There is a typo in line 160 of get_GSOD():

• load(system.file("extdata", "country_list.Rda", package = "GSODR"))
• Rda should be rda

Resulting in the following error:

Error in readChar(con, 5L, useBytes = TRUE) : cannot open the connection
In addition: Warning message:
In readChar(con, 5L, useBytes = TRUE) :
cannot open compressed file '', probable reason 'No such file or directory'

R version 3.4.1 (2017-06-30) -- "Single Candle"
Copyright (C) 2017 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

Typing the first example of the get_GSOD's help produces an error:

> t <- get_GSOD(years = 2010, station = "955510-99999")
Error in .f(.x[[i]], ...) : 
955510-99999 is not a valid station ID number, please check your entry.
Valid Station IDs can be found in the isd-history.txt file
available from the US NCEI FTP server by combining the USAF and WBAN
columns, e.g., '007005' '99999' is '007005-99999' from this file 
<ftp://ftp.ncdc.noaa.gov/pub/data/noaa/isd-history.txt>

@thengl, I don't know what's happened, but your edits to the paper.md file have gone missing and I cannot find them in git anywhere.

I moved the /paper directory to /inst/paper and I think perhaps you were editing the version in the previous location and the changes got clobbered with the moving directories around?

Can you check and update your information and edits in the current location of the file? I see that there is a .png file in the /paper directory. That directory should not exist in the current master or devel branches.

This value should only be a positive number equal to or larger than 1.

If max-missing is specified for an incomplete year, e.g. download 2017 in November 2017, it will omit all stations and data.

Check to see if function is downloading current year and warn user that using max-missing is not possible for current year data.

Split the spatial work into a separate vignette, keep main examples in main vignette to cut down on size.

@sckott,
I'm unable to switch on the Travis to build this package since I'm not an administrator any longer now so the build fails.

I've updated the links in the documentation. Just need to fix things up on the Travis side of things, I think?

So, this is a strange issue which I'm not sure how to diagnose.
reformat_GSOD() does not behave as expected. It runs, but with no end, no output, no and CPU/GPU activity shown on the activity monitor by Mac. It's like nothing is happening at all.

I narrowed everything down to one single file, but nothing.

library(GSODR)

list.files("data/gsod/2010/")
[1] "02431099999.csv"

data <- reformat_GSOD(dsn = "data/gsod/2010")

The file located at ftp://ftp.ncdc.noaa.gov/pub/data/noaa/isd-history.csv has incorrect station altitudes.

Correct using DEM as best possible.

I may be confusing something here, but I think that a while back, reformat_GSOD() worked with *.op.gz files obtained from NOAA ftp.
It seems like this is no longer the case. Is something changed?

This is not an issue per se, it's a confusion. I couldn't rerun my code from a year ago, and I'm trying to see what changes do I need to make to the code to make it work.

I tried running the following code:

library(GSODR)
g <- get_GSOD(years='2016', country='NZ', path=tempdir())

And I received the error message:

Error in { : task 1 failed - "task 1 failed - "unrecognized date format""

Below is my sessionInfo():

R version 3.3.1 (2016-06-21)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 14.04.5 LTS

locale:
 [1] LC_CTYPE=en_AU.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_AU.UTF-8        LC_COLLATE=en_AU.UTF-8    
 [5] LC_MONETARY=en_AU.UTF-8    LC_MESSAGES=en_AU.UTF-8   
 [7] LC_PAPER=en_AU.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_AU.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] dplyr_0.5.0 GSODR_0.1.9

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.7       lattice_0.20-33   codetools_0.2-14  foreach_1.4.3    
 [5] assertthat_0.1    chron_2.3-47      grid_3.3.1        R6_2.1.3         
 [9] DBI_0.5-1         magrittr_1.5      settings_0.2.4    stringi_1.1.1    
[13] curl_2.1          data.table_1.9.6  doParallel_1.0.10 raster_2.5-8     
[17] sp_1.2-3          iterators_1.0.8   tools_3.3.1       stringr_1.1.0    
[21] parallel_3.3.1    compiler_3.3.1    tibble_1.2  

EDIT: This was with the CRAN version of GSODR

Is there a way to do batch processing, if I have multiple stations. I have tried using a vector and having the station[1] in the get_gsod, which it did not take and I also tried using the paste command but was getting this error: We've tried to get the file(s) you requested six times, but the server is not responding, so we are unable to process your request now. Please try again later.

Thanks,

Shaffiq

Often there are several missing min/max temperature values or stations missing several days of complete sets of data.

Several options exist for interpolating missing station data using within and between station data.

Add ability to interpolate missing values for temperature, min temperature and max temperature.

Remove CSV and GeoPackage writing from this function.

There has been some slippage with changes in the code over time. It's time to improve the test coverage again before releasing a new version.

Hi,

When i'm trying to get the country_lsit with this code:

load(system.file("extdata", "country_list.rda", package = "GSODR"))

R shows me the next error:

Error in readChar(con, 5L, useBytes = TRUE) :
no se puede abrir la conexión
Además: Warning message:
In readChar(con, 5L, useBytes = TRUE) :
cannot open compressed file '', probable reason 'Invalid argument'

Any idea what is going wrong? i'm trying the same with:

load(system.file("extdata", "isd_history.rda", package = "GSODR"))

and is working fine..

Thanks in advance!

The precipitation values appear lower than expected/observed and some of data are also missing. Is there anything I can do about these low values? How do I reduce the number of values removed from the plots?

Data directory is too large for CRAN

T. Hengl's original script also generates a shape file.

Incorporate this functionality as an option into the get_GSOD function so that the use can select a CSV (default output), shape file (optional) or both (optional).

Just tried your function using the following setting:

GSODR::get_GSOD(years = 1973:2015, path = "~/Downloads/", station = "080250-99999")

Failing with the error above. If I set years to 2003:2015, everything works.

Thanks for checking!

Final output should have these values in them. Currently, they do not appear but it is in the internal database file that GSODR distributes.

RH is sometimes calculated at being >100%. This means that ea and es are also incorrect.

Check TDEW and TEMP relationships in these data to see what relationship exists in these instances and correct or remove spurious data.

From the text file header description:

THIS INVENTORY SHOWS THE NUMBER OF WEATHER OBSERVATIONS BY STATION-YEAR-MONTH FOR BEGINNING OF RECORD THROUGH SEPTEMBER 2017. THE DATABASE CONTINUES TO BE UPDATED AND ENHANCED, AND THIS INVENTORY WILL BE UPDATED ON A REGULAR BASIS.

ftp://ftp.ncdc.noaa.gov/pub/data/noaa/isd-inventory.txt

The item, tf, should be set as a 'tempfile() in get_GSOD.

See line 209 of https://github.com/adamhsparks/GSODR/blob/master/R/get_GSOD.R.
tf <- "~/tmp/GSOD-2010.tar should instead be, tf <- tempfile()

This will make using the GSODR package faster to use since repeated requests for this file will not be necessary.

When a specific country is selected for download and cleaning an error is generated, data are not downloaded.

Allow users to select whether inconsistent Lat/Lon, MIN/MAX values are discarded or kept.

Keep current behaviour as default, discard or set to NA, but allow for keeping values if desired.

See reformat_GSOD() source code below. Object file_list is an input parameter, but it is reassigned automatically when dsn is not NULL, and input value is ignored. I suppose file_list is only used when input dsn is NULL, but that is not clear in the package documentation, and the required format for file_list is also not clear from documentation. Thank you for the package!

Alison

reformat_GSOD <- function (dsn = NULL, file_list = NULL)
{
isd_history <- NULL
load(system.file("extdata", "isd_history.rda", package = "GSODR"))
setkeyv(isd_history, "STNID")
if (!is.null(dsn)) {
file_list <- list.files(path = dsn, pattern = "^.*\.csv$",
full.names = TRUE)
if (length(file_list) == 0)
stop("No files were found, please check your file location.")
}
GSOD_XY <- .apply_process_csv(file_list, isd_history)
return(GSOD_XY)
}

Users can enter any value for a station, there are no checks as for years or file path for validity and/or existence in database