whatNWISsites(stateCd="OH",parameterCd=c("00665", "00060"))
URL caused an error:http://waterservices.usgs.gov/nwis/site/?format=mapper&stateCd=OH&parameterCd=0066500060

Use case: I want a list of sites that have param X and param Y

The daily streamflow data for "05270500" contain 2 codes for 2014-11-11. An extract of the WML is below.

<ns1:value qualifiers="P" dateTime="2014-11-10T00:00:00.000">315/ns1:value
<ns1:value qualifiers="P Ice" dateTime="2014-11-11T00:00:00.000">-999999/ns1:value
<ns1:value qualifiers="P Bkw" dateTime="2014-11-11T00:00:00.000">-999999/ns1:value
<ns1:value qualifiers="P Ice" dateTime="2014-11-12T00:00:00.000">-999999/ns1:value
<ns1:value qualifiers="P Ice" dateTime="2014-11-13T00:00:00.000">-999999/ns1:value
<ns1:value qualifiers="P Ice" dateTime="2014-11-14T00:00:00.000">-999999/ns1:value

When the data are retrieved though 2014-11-10, or beginning on 2-014-11-12, no error/warning is printed and the data look OK. If the retrieval overlaps that period, a weird message is printed and the data are coded as 1/0. See the example retrieval:

readNWISdv("05270500", "00060", "2014-11-10", "2014-11-12")
Aggregation function missing: defaulting to length
agency_cd site_no Date X_00060_00003_cd X_00060_00003
1 USGS 05270500 2014-11-10 1 1
2 USGS 05270500 2014-11-11 2 0
3 USGS 05270500 2014-11-12 1 0

http://waterservices.usgs.gov/rest/IV-Service.html#Specifying

http://waterservices.usgs.gov/rest/Statistics-Service-Test-Tool.html

From an email:
""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Back in January, I tried using getNWISDaily to pull groundwater water quality data and to my surprise (at the time) it worked! However, in trying this stunt again today, it didn't work, which has me wondering if 1) getNWISDaily has been modified in such a way over the last couple of months to temporarily disable this functionality or 2) it was a fluke that it worked in the first place? For example, gw site 390208119433201 and param code 00631 (nitrate) has data (see screen shot below), but the following script won't seem to pull the data. Is there a new function for pulling gw data?

Thanks for your consideration. Best, Eric

library(EGRET)
library(dataRetrieval)

StartDate <- ""  
EndDate <- ""
siteNumber <- "390208119433201"
QParameterCd <- "00631"
Daily <- readNWISDaily(siteNumber, QParameterCd, StartDate, EndDate)

proof that the data is there:

http://nwis.waterdata.usgs.gov/nwis/qwdata?site_no=390208119433201&agency_cd=USGS&format=serial_rdb

From Anne @ NM Training.

I received an error the other day about "pcodes may be mistyped...". Unfortunately, I did not save the exact error. However, pcodes were retrieved using whatNWISdata so they were not mistyped. I was pulling all data for a site, and the "mistyped" pcodes looked like something from a pesticide schedule.

I will make sure to save the error next time with exact pcodes.
The code I am using to do this data pull is in the QWToolbox package under the publicData function.

Hi there-

First of all, this tool is fantastic. Thank you for making it.

The discovery aspect of this tool is great and critical to my work. But I'm a bit confused about the output of this kind of command meta <- whatWQPsites(huc='10190005'), which gives me meta$MonitoringLocationIdentifier like this

...
[565] "NARS_WQX-FW08CO004" ...
[583] "USGS-06725100"      ...
[613] "USGS-395528105133800" ...

in relationship to use of readNWISSuv. Can I only supply the numeric part of the MonitoringLoactionIdentifiers for rows/sites with identifiers like row 583 as the siteNumbers argument? The documentation is not specific ( siteNumbers character USGS site number (or multiple sites). This is usually an 8 digit number ) other than possibly limiting the query to USGS data. But the queries using other identifiers, even the longer USGS ones, have failed for me. In other words, am I only able to get a small subset of the data listed in meta with readNWISSuv and is there a way to get the other data listed in meta? Maybe I'm missing something.

Also, by the way, is there any access to gages-II data/attributes via these NWIS queries?

Thanks again,
james

wqp_DF1 <- readWQPdata(statecode="US:20", siteType="Stream", sampleMedia="Water", startDateLo="01-01-2014",characteristicType="Nutrient" )
min(wqp_DF1$ActivityStartDate)
[1] "1916-06-11"

Seems to be an issue since this last round of commits from earlier this week.

It would be fantastic to have a function to convert state codes (or county codes) to character names.

Seems that the generic readNWISdata.r isn't called by the other functions that take specific args. Could reduce the code to pass the args into the generic function to build the URLs and deal with the responses in one location. Thoughts?

https://github.com/USGS-R/dataRetrieval/blob/master/R/importWQP.R#L67

Something is a bit confusing here:

> d = readNWISuv('01670080','62620',startDate='2015-03-19',endDate='2015-03-19')
> head(d)
  agency_cd  site_no            dateTime tz_cd X_62620_00011_cd X_62620_00011
1      USGS 01670080 2015-03-19 04:00:00   EDT                P         -0.86
2      USGS 01670080 2015-03-19 04:06:00   EDT                P         -0.95
3      USGS 01670080 2015-03-19 04:12:00   EDT                P         -1.03
4      USGS 01670080 2015-03-19 04:18:00   EDT                P         -1.10
5      USGS 01670080 2015-03-19 04:24:00   EDT                P         -1.19
6      USGS 01670080 2015-03-19 04:30:00   EDT                P         -1.29
> format(d$dateTime[2],usetz=TRUE)
[1] "2015-03-19 04:06:00 UTC"
> format(d$dateTime[2],tz='America/New_York',usetz=TRUE)
[1] "2015-03-19 00:06:00 EDT"

From the printout It appears to be doing the request for a GMT-bounded day, but the timestamps on the actual data appear to be for for a local-standard-time bounded day. If it is the latter, then this printout is a bit misleading. Since the time data seems to be translated into UTC, maybe the data returned shouldn't have the timezone column.

In the '?readNWISuv' help, it might be nice to highlight that the startDate and endDate are in the station local-standard time, not the date ranges indicated in http://waterservices.usgs.gov/rest/IV-Service.html#Specifying

For reference:
> time2=as.POSIXct(360, origin = "2015-03-19", tz = "GMT")
> format(time2,tz='GMT',usetz=TRUE)
[1] "2015-03-19 00:06:00 GMT"
> format(time2,tz='America/New_York',usetz=TRUE)
[1] "2015-03-18 20:06:00 EDT"

It would be great to be able to specify how these functions deal with 1) no internet connect 2) no matching site 3) empty returns. Right now I wrap the calls in tryCatch, but what is your take on handling these cases inside the functions?

what can be done about this? seems like a problem on NWIS's side.
It's fairly annoying that this is not 1-1... but manageable.

dataRetrieval::readNWISsite("02054530")

  agency_cd  site_no                         station_nm site_tp_cd lat_va long_va dec_lat_va dec_long_va
1      USGS 02054530       ROANOKE RIVER AT GLENVAR, VA         ST 371604  800823   37.26791   -80.13949
2     VA087 02054530 ROANOKE RIVER AT GLENVAR, VIRGINIA         ST 371604  800823   37.26791   -80.13949
  coord_meth_cd coord_acy_cd coord_datum_cd dec_coord_datum_cd district_cd state_cd county_cd country_cd
1             M            U          NAD27              NAD83          51       51       161         US
2             M            U          NAD27              NAD83          51       51       161         US
  land_net_ds  map_nm map_scale_fc alt_va alt_meth_cd alt_acy_va alt_datum_cd   huc_cd basin_cd topo_cd
1             GLENVAR        24000   1100           L       0.01       NGVD29 03010101                 
2             GLEMVAR        24000   1100           L       0.01       NGVD29 03010101                 
                  instruments_cd construction_dt inventory_dt drain_area_va contrib_drain_area_va tz_cd
1 NNNNYNNNNNNNNNNNNNNNNNNNNNNNNN                                        281                    NA   EST
2 NNNNYNNNNNNNNNNNNNNNNNNNNNNNNN                                        281                    NA   EST
  local_time_fg reliability_cd gw_file_cd nat_aqfr_cd aqfr_cd aqfr_type_cd well_depth_va hole_depth_va
1             Y                  NNNNNNNN                                             NA            NA
2             Y                                                                       NA            NA
  depth_src_cd project_no
1                        
2                        

Causing errors at CRAN.

I got a warning from

Well <- "364200119420001"
Pcodes <-c("00010", "00095", "00300", "00400")

TestWellQW <- readNWISqw(Well, Pcodes, expanded=FALSE, reshape=TRUE)
It was:
Warning message:
In readNWISqw(Well, Pcodes, expanded = FALSE, reshape = TRUE) :
Reshape can only be used with expanded data. Reshape request will be ignored.

But the data were reshaped and it does in fact make sense to reshape the data when expanded=FALSE (maybe even more so than when expanded is TRUE).

Linked issue from EflowStats issue number 17 .

This works:

library(dataRetrieval)
x_obs <- readNWISdv("02177000", "00060", "2009-10-01", "2013-09-30") 

This does not, returns NAs

library(dataRetrieval)
library(reshape)
x_obs <- readNWISdv("02177000", "00060", "2009-10-01", "2013-09-30") 

Thought pCodeToName would let me do something like pCodeToName('00300'), but it is just a big-ol data.frame.

We are looking for basic support for units(pcode) or something similar. Any help for the user here?

In the past I have used the waterData package (http://cran.r-project.org/web/packages/waterData/index.html) to access the USGS Daily Values Site Web Service. However, waterData is not listed among the USGS-R repositories. Does the dataRetrieval package supersede the waterData package? If not, are there any documents that provide guidance on the differences between these packages?

On Windows 7, R

 getNWISData(stateCd='FL', parameterCd="00010")
Error in read.table(file = file, header = header, sep = sep, quote = quote,  : 
  more columns than column names 

R version info

platform       x86_64-w64-mingw32          
arch           x86_64                      
os             mingw32                     
system         x86_64, mingw32             
status                                     
major          3                           
minor          1.0                         
year           2014                        
month          04                          
day            10                          
svn rev        65387                       
language       R                           
version.string R version 3.1.0 (2014-04-10)
nickname       Spring Dance         

EDITED AS OF dataRetrieval 2.4.0 and greater:
Within a USGS firewall, it is possible to retrieve data flagged as 'internal' by using the setAccess function:

library(dataRetrieval)
setAccess('internal')
data <- readNWISuv("373015122071000","00095")

Original message:
Within a USGS firewall, it is possible to retrieve data flagged as 'internal' using "Access=3". Using the url attribute, that can be done as follows:

library(dataRetrieval)
url <- "http://nwis.waterdata.usgs.gov/ca/nwis/uv?cb_00095=on&cb_00095=on&format=rdb&site_no=373015122071000&period=&begin_date=2013-03-06&end_date=2015-07-07&Access=3"
DS <- importRDB1(url, asDateTime=TRUE)

findData <- whatNWISdata("373015122071000")
nrow(findData)
[1] 26
urlToFind <- attr(findData, "url")
urlToFind <- paste0(urlToFind, "&Access=3")
findData2 <- importRDB1(urlToFind)
nrow(findData2)
[1] 31

Depending on the function you are initially trying to retrieve data, you'll want to check if the import is RDB (tab delimited) or WaterML1 and replicate what was happening in the original convenience function. This can be done by either inspecting the function:

readNWISdv
function (siteNumber,parameterCd,startDate="",endDate="",statCd="00003"){  

  url <- constructNWISURL(siteNumber,parameterCd,startDate,endDate,"dv",statCd=statCd)

  data <- importWaterML1(url, asDateTime=FALSE)
  if(nrow(data)>0){
    data$dateTime <- as.Date(data$dateTime)
    data$tz_cd <- NULL

    names(data)[names(data) == "dateTime"] <- "Date"    
  }


  return (data)
}
<environment: namespace:dataRetrieval>

So, for the readNWISdv function, add the "&Access=3" to the url, call importWaterML1, and convert the dateTime to a Date:

siteNumber <- '04085427'
startDate <- '2012-01-01'
endDate <- '2012-06-30'
pCode <- '00060'
rawDailyQ <- readNWISdv(siteNumber,pCode, startDate, endDate)
newURL <- attr(rawDailyQ, "url")
newURL <- paste0(newURL,"&Access=3")
newData <- importWaterML1(newURL, asDateTime=FALSE)
newData$dateTime <- as.Date(newData$dateTime)
newData$tz_cd <- NULL
names(newData)[names(newData) == "dateTime"] <- "Date"   

Alternatively, the readNWISdata could be used. For the example above:

siteNumber <- '04085427'
startDate <- '2012-01-01'
endDate <- '2012-06-30'
pCode <- '00060'

readNWISdata(service = "dv", sites=siteNumber, parameterCd=pCode, 
             startDate=startDate, endDate=endDate, Access=3)

Although the official arguments are ..., getting at least the 'major' filters in the description would be super useful.

I have a site where one sensor is discontinued for another. These were assigned different DD numbers and returned in separate columns by readNWISuv. Is there an easy way in R to merge the two into one column filling in the missing values? Here is an example of such a site:

http://waterservices.usgs.gov/nwis/iv/?format=rdb&sites=07144100&startDT=2015-02-01&endDT=2015-04-01&parameterCd=63680

In this case, fill 96_63680 when 44_63680 is missing a value.

Thanks,
Steve

How to reproduce: have another package import dataRetrieval and use one of its functions inside the other package function. But don't have dataRetrieval attached (see screen shot). If you have that box checked, all is good.

Improve efficiency by reading line by line, getting comments and data at the same time.

using (in roxygen comments):

'@Seealso \link{getSiteFileData}, \link{getDataFromFile}

for examples of similar functions or functions that are also in the #' @examples section. Makes it easier to browse and get to where you want to be.

Is there a reason that readNWISpeak fails when retrieving this data

readNWISpeak("07154400")

Data seem fine from NWIS

http://nwis.waterdata.usgs.gov/usa/nwis/peak/?site_no=07154400&range_selection=date_range&format=rdb

siteNumber vs sites appears in different functions. Should consider unifying this so it is easy to switch between functions and you have a expectation of the design pattern.

This info seems to be provided by waterML, but isn't distributed to the user. NA is a pretty common way to convey such info in R and results in more meaningful error handling (for example, average of a list of values with NA in it is NA, but -999999 averages in just fine).

It seems that if there is no NWIS data available, the user gets a confusing error

URL caused an error: http://waterservices.usgs.gov/nwis/dv/?site=10264750&format=rdb,1.0&ParameterCd=00060&StatCd=00003&startDT=1980-01-01&endDT=2014-11-03
Content-Type=text/plain; charset=UTF-8
Error in names(data) <- c("agency", "site", "dateTime", "value", "code") : 
  attempt to set an attribute on NULL

To reproduce:

getNWISDaily("10264750","00060", "1980-01-01", "2014-11-03", convert=TRUE)

Tried it with the fork at @ldecicco-USGS. Using:

 readNWISdv("10264750","00060", "1980-01-01", "2014-11-03")

get

Error in timeSeries[[i]] : subscript out of bounds

Not sure if its the same error or not, but I think it is.

Add in NWIS parameters groups as selection options for pCodes. At a minimum, allow for pull of all measured pCodes at a site.

Similar to the recent upgrades to readWQPdata, allow R dates in startDT, numbers in bBox, etc.

Check headers...to improve user experience

users have reported lost data from importWQP

Hi,

The function inputs listed for the function readNWISuv() on the main dataRetrieval page are incorrect. The inputs are listed as "Common 3, parameter code". The help for the function lists the arguments as ""siteNumbers, parameterCd, startDate = "", endDate = "",
tz = "". Please update the main page - I had issues getting the correct data output the first few times I tried.

This works:

nutrient <- readWQPdata(statecode="US:55",characteristicType="Nutrient",
                         siteType="Lake,+Reservoir,+Impoundment")

But this doesn't:

nutrient <- readWQPdata(statecode="US:55",characteristicType="Nutrient",
                         siteType="Lake%2C+Reservoir%2C+Impoundment")

which is what you'd copy from the WQP itself.

I am trying to get the state code for a bunch of sites, and am looping through them and using readNWISsite(nwis_site). This has always worked out fine, but now is giving me a bad connection error after a few hundred of them. Seen this before?

Little/no info exists about startDateTime/endDateTime and what it is.