16eagle / getspatialdata Goto Github PK

Do you know why getMODIS_names is an empty list ?

getMODIS_names(username = "xxxx")
list()
product <- grep("MOD13Q1", product_names, value = T)
product
character(0)

And when I try to create a query, the dataset isn't available. Is it the name of the product which doesn't work ?

query = getMODIS_query(time_range = c("2018-01-01","2017-10-25"), name = "MOD11A1", aoi = coordinates)
Searching USGS EarthExplorer for available products...
Error: Invalid query. This dataset seems to be not available for the specified time range.

Thank you.

Originally posted by @GwMo in #3 (comment)

getLandsat_data fails when sr_ndvi is requested. The order gets submitted to espa, but the function failes.
Works with "sr" level.

files <- getLandsat_data(records = query_filtered, level = "sr_ndvi", source = "auto")
Ordering requested items from ESPA...
Collecting from 1 collection(s) [olitirs8_collection], resulting in 1 order(s)...
Products 'LC08_L1TP_19....'.....01_T2' have been ordered successfully:
[level = 'sr_ndvi', format = 'gtiff', order ID(s) '[email protected]'].
Error in xj[i] : invalid subscript type 'list'

I'm trying to download sentinel 2 images with the package, and it works perfect until the getSentinel_data function. However, when executing the prepSentinel function,

datasets_prep <- prepSentinel(datasets, format = "tiff")

I get the following error:

Error in gdal_chooseInstallation(hasDrivers = of) : No installations match. Además: Warning messages: 1: In system(cmd, intern = TRUE) : comando ejecutado '"C:\Program Files\QGIS 3.10\bin\gdalbuildvrt.exe" -resolution "highest" -separate

Thank you very much in advance

Thanks for the great package, really saves time to filter and download EO data!

I think I found a small problem in the getSentinel_query function. I see that I always get 200 records of the returned data.frame.
In line 74 in getSentinel_query function (I downloaded github version 28 May 2018),
the following lines will quit the while loop after the second batch of requests:

if (length(query.list) != 100) re.query <- F
The query.list list is progressively filled with results so I believe it could be changed to:

if (length(query.list) != row.start+100 ) re.query <- F

Hope this helps.
Cheers,
Francesco

PS an optional input parameter to the function, defining maximum number of returned results would also be a very nice plus, to avoid infinite number of downloaded records.

prepSentinel extraction fails on Windows for directories with large path lengths. Same issue occurs for renaming folders in such directories.

Getting this error (intermittently I think) when running getLandsat_query. I've traced it back to .EE_query in Internal.R

Here is the code I use:

  # Set start and end date for cloud cover (as far back as GLAD alerts go to current day)
  start_date <- paste0(year-num_years, "-01-01")
  end_date <- as.character(Sys.Date())
  
  # Query USGS database for all Landsat scene metadata in our area
  query <- getLandsat_query(c(start_date, end_date),
                            name="all",
                            aoi=remove.holes(aoi_sp),
                            verbose=TRUE
  )

For every record handed to getLandsat_data, two individual orders instead of one are placed at the same time.

The install says something along the lines of "no GDAL installed" and then provides a list of gdal providers, though it doesn't say where the file should be placed for the install to work, nor does it mention on the intstall page of this Github repository that it needs to be installed as a requirment to begin with.

Please help!

Thanks in advance,

-Russell Gray

Hi - Just came across your work on getSpatialData. Looks like a package that will have great use for lots of folks.

As I was looking through the readme, I realized this might be a great application for the mapedit package, in which users can interactively draw their own areas of interest and such.

Just figured I'd put the idea forth in case you weren't aware of or hadn't thought of mapedit for this. But thanks for your time on this!

Hello again,
if I am not mistaken, the current version of the package is capable to download Sentinel 3 data but not to further process them (e.g. using prepSentinel function).

I was just wondering whether there is a plan to implement options for processing of Sentinel 3 data which are in the ncdf format. I am especially interested in LST data but also some bands from OLCI. I am unfortunately not able to modify/develop the codes for that purpose at this moment but I am keen to test them if they become available in the future.

All the best,
Milan

getSentinel_data() was running overnight, and it failed sometime during download of the sixth record (the first 5 records appear to have downloaded correctly). After restarting the process at record 7
datasets <- getSentinel_data(records = records_filtered[7]) it returns the following Error: 'records$uuid' needs to be of type 'character'. class(records_filtered$uuid) returns "character" . I tried getSentinel_data() on various indices and all return the same error. I have restarted R and re-installed getSpatialData with the same result. Running R 3.5.2 on Windows 10.

I got two problems when I tried to run the code from this tutorial(I only used data from USGS): https://github.com/16EAGLE/getSpatialData/blob/master/README.md

(1) When I run view_previews(records), got the error
"Error in leafem::addHomeButton(map = map, ext = ext, layer.name = layer.name, :
unused argument (layer.name = layer.name)".

(2) When I run calc_cloudcov(records), got the error
[Aoi cloudcov calc 1/1] Processing: LC08_L1TP_194025_20200516_20200527_01_T1
Error in st_write.sf(records, dsn = file, append = append, quiet = TRUE) :
unrecognized argument(s) append

Error: Failed to write file: mypath/LC08_L1TP_194025_20200516_20200527_01_T1.geojson

Many thanks if anyone could help with these two problems.

First - thanks @16EAGLE for your work on this package!

I'm trying to use getSpatialData to pull MODIS data but when running the getMODIS_data function, I'm getting a bad request error. Here's my code:

library(getSpatialData)
library(sp)
library(raster)

set_archive("D:/lcdata_explore", create = TRUE) 

login_USGS(username = paste(username), password = paste(password))

modNames <- getMODIS_names()

landcover.prod <- grep("MCD12Q1", modNames, value = T)

y_min <- 40.578238
x_max <- -109.233825
y_max <- 47.05
x_min <- -115.838981

e <- as(raster::extent(x_min, x_max, y_min, y_max), "SpatialPolygons")
proj4string(e) <- "+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs"
set_aoi(e)

view_aoi()


time_range <-  c("2015-01-01", "2015-12-31")

query <- getMODIS_query(time_range = time_range, name = landcover.prod)

getMODIS_preview(record=query[2,])

files <- getMODIS_data(query[2,])

The resulting error is:

Starting download of product(s) 'MCD12Q1.A2015001.h10v04.006'.
Attempting to download 'MCD12Q1.A2015001.h10v04.006.2018055062901.hdf' to 'D:/lcdata_explore/get_data/MODIS//MCD12Q1.A2015001.h10v04.006.2018055062901/MCD12Q1.A2015001.h10v04.006.2018055062901.hdf'...
Downloading: 95 kB     Error in gSD.get(url.file, dir.file = file.tmp, prog = T) : 
  Bad Request (HTTP 400). Failed to connect to server..

Here's my sessionInfo():

R version 3.5.1 (2018-07-02)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 17134)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] raster_2.8-19        sp_1.3-1             getSpatialData_0.0.4

loaded via a namespace (and not attached):
 [1] remotes_2.0.2      sf_0.7-3           lattice_0.20-35    mapview_2.6.3     
 [5] colorspace_1.4-0   viridisLite_0.3.0  usethis_1.4.0      htmltools_0.3.6   
 [9] stats4_3.5.1       yaml_2.2.0         base64enc_0.1-3    rlang_0.3.1       
[13] pkgbuild_1.0.2     e1071_1.7-0.1      R.oo_1.22.0        later_0.8.0       
[17] DBI_1.0.0          glue_1.3.0         withr_2.1.2        R.utils_2.8.0     
[21] sessioninfo_1.1.1  foreach_1.4.4      munsell_0.5.0      mapedit_0.4.3     
[25] R.methodsS3_1.7.1  devtools_2.0.1     htmlwidgets_1.3    codetools_0.2-15  
[29] memoise_1.1.0      callr_3.1.1        httpuv_1.4.5.1     ps_1.3.0          
[33] crosstalk_1.0.0    curl_3.3           class_7.3-14       gdalUtils_2.0.1.14
[37] Rcpp_1.0.0         xtable_1.8-3       classInt_0.3-1     backports_1.1.3   
[41] promises_1.0.1     scales_1.0.0       satellite_1.0.1    desc_1.2.0        
[45] pkgload_1.0.2      jsonlite_1.6       webshot_0.5.1      leaflet_2.0.2     
[49] mime_0.6           fs_1.2.6           png_0.1-7          digest_0.6.18     
[53] processx_3.3.0     shiny_1.2.0        getPass_0.2-2      grid_3.5.1        
[57] rprojroot_1.3-2    rgdal_1.4-2        cli_1.0.1          tools_3.5.1       
[61] magrittr_1.5       crayon_1.3.4       xml2_1.2.0         prettyunits_1.0.2 
[65] assertthat_0.2.0   httr_1.4.0         rstudioapi_0.9.0   iterators_1.0.10  
[69] R6_2.4.0           units_0.6-2        compiler_3.5.1    

Hi,

This question I believe was also raised here.

https://stackoverflow.com/questions/58930318/don-work-prepsentinel-with-sentinel-1-data

Do you have a solution or explanation to this? Is the data now repackaged and this function no longer current for sentinel-1?

Hi there,
Thanks for developing such a useful tool. I am currently tasked with querying the HDDS website - Hazards Data Distribution System - https://hddsexplorer.usgs.gov/, which is part of USGS Earth Explorer and uses the same login information. Are there additional functions or code that are required to access this portal? Any help or advice would be great.

I'm following the example of the documentation but in this case with Landsat data.

The problem is in this part of the code:

makeCluster(detectCores() - 1) %>% registerDoParallel(cl)

foreach(i = 1:nrow(records[]), 
             .combine=c, 
             .packages='getSpatialData') %dopar% {
             getLandsat_data(records[i, ], dir_out = outdir)
         }
#### stop cluster ####
stopImplicitCluster()

I get the following error:
"Error: Argument 'username' needs to be of level 'character'. You can use 'login_USGS()' to define your login credentials globally."

I signed in into the USGS account, I tried the normal download and it works!
Also, I tried the next (Added the user and password explicy):

foreach(i = 1:nrow(records[]), 
             .combine=c, 
             .packages='getSpatialData') %dopar% {
             getLandsat_data(records[i, ], dir_out = outdir,
             username = "my_user", password = "my_pass")
         }

But it neither works.

Any comments or suggestions are welcome.

Congratulations on the package!

Tried loading a shapefile and setting it as an aoi but failed
aoi <- as_Spatial(st_read("./Boundary.shp"))

Here is the erro
Error: Argument 'aoi' needs to be a 'SpatialPolygons' or 'sfc_POLYGON' or 'matrix' object.

It would be nice if you could see in the console the number of the current downloading scene and the number of all scenes, i. e. paste0("Downloading scene ", iterator " of ", length(records), ".".

I'm getting the following error on R 3.6.1 using R studio 1.1.456: Error: Could not process request: is.character(user) is not TRUE

Here's the code used:
getSentinel_query(platform = "Sentinel-5p", aoi = EA_shp, time_range = c("2020-01-01", "2020-02-28"), username = "s5pguest", password = "s5pguest")

I try to download images from the MODIS satellite (MODIS_MYD09GA_V6), but when I apply the getMODIS_data command I only get a 4.6 kb file. The true full database has on average 50 mb.
Sorry my english.

Thanks in advance,

I think there should be additional arguments in the function getSentinel_data() - unzip and file.remove if you don't want zip archives but only unzipped catalogs.

Hi,

Thanks for your very good work, I am very excited to use your package to make transparent and reproducible science !

An interesting feature would be to be able to retrieve a product metadata table (i.e. output of getxxx_query) by providing its unique identifier (or a set of identifiers) as input parameter of the function, i.e. without providing a time frame and/or a ROI.

Cheers
Paul

Attempting to get_data gives me a 404 for one record but not the other. I'm having trouble tracking down the troublesome URL being constructed in get_data. The second record is accessed just fine. Thanks for the help!!

Code:

Locations = data.frame(39.9,-105.65)
colnames(Locations)=c("Lat","Lon")

cell_dim = .1
Rects$BR_Lat = Locations$Lat
Rects$BR_Lon = Locations$Lon
Rects$TR_Lat = Rects$BR_Lat+cell_dim
Rects$TR_Lon= Rects$BR_Lon
Rects$TL_Lat= Rects$TR_Lat
Rects$TL_Lon= Rects$TR_Lon - cell_dim
Rects$BL_Lat= Rects$BR_Lat
Rects$BL_Lon=Rects$TL_Lon

a<-matrix(as.numeric(as.character(Rects[1, ])), ncol=2, byrow=TRUE)

set_aoi(a)
view_aoi() #view AOI in viewer, which will look like this:

start_records = "2020-06-01"
end_records = "2020-07-05"

#records = get_records(time_range = c(start_records, end_records), products = c("LANDSAT_8_C1", "Sentinel-2", "LANDSAT_ETM_C1"))
records = get_records(time_range = c(start_records, end_records), products = c("LANDSAT_8_C1"))

records_filtered <- records[which(records$level %in% c("l1")),] #filter by Level
records_filtered <- records_filtered[as.numeric(records_filtered$cloudcov) <= 70, ] #filter by clouds

#order any data that is not already available 
test_for_availability = check_availability(records_filtered)

tryCatch(records_filtered <- order_data(test_for_availability), error = function(e) message("No New Records Ordered: ", e))

#test for all data availability for 60 minutes else proceed with current data
time_waited=0
cycle=60

#try for an hour before just going with all avaialble now
while(!all(test_for_availability$download_available==TRUE && time_waited<3600)){
  Sys.sleep(cycle)
  test_for_availability = check_availability(records_filtered)
  time_waited=time_waited+cycle
}

records_filtered = test_for_availability[test_for_availability$download_available==TRUE,]
records_filtered <-records_filtered[order(records_filtered$cloudcov),]

get_data(records = records_filtered[1,])
get_data(records = records_filtered[2,])

Output:

get_data(records = records_filtered[1,])
Receiving MD5 checksums...
Assembling dataset URLs...
Error in .get(index_url) :
Not Found (HTTP 404). Failed to process request.
get_data(records = records_filtered[2,])
Receiving MD5 checksums...
Assembling dataset URLs...
[Dataset 1/1 | File 1/38] Skipping download of dataset 'LC08_L1TP_034032_20200616_20200626_01_T1 (l1)', since '/mnt/Backup/Spatial/_datasets/LANDSAT_8_C1//LC08_L1TP_034032_20200616_20200626_01_T1/LC08_L1TP_034032_20200616_20200626_01_T1_B6.TIF' already exists...
[Dataset 1/1 | File 2/38] Skipping download of dataset 'LC08_L1TP_034032_20200616_20200626_01_T1 (l1)', since '/mnt/Backup/Spatial/_datasets/LANDSAT_8_C1//LC08_L1TP_034032_20200616_20200626_01_T1/LC08_L1TP_034032_20200616_20200626_01_T1_B11.TIF' already exists...

This is probably a regression. Previously, Sentinel 2 SAFE format was downloaded normally, but now is without .zip extension and there is no possibility to unzip (you must add it manually).

library("getSpatialData")
login_CopHub(username = "username", password = "password")
data("aoi_data")
aoi <- aoi_data[[1]]
set_aoi(aoi)

records <- getSentinel_query(time_range = c("2019-01-01", "2019-01-30"), platform = "Sentinel-2")

datasets <- getSentinel_data(records[1, ], dir_out = "C:/scene")
datasets
[1] "C:/scene/S2A_MSIL2A_20190127T102311_N0211_R065_T32UPV_20190127T134739.SAFE"

list.files("C:/scene")
[1] "S2A_MSIL2A_20190127T102311_N0211_R065_T32UPV_20190127T134739.SAFE"

Hi,

after updating packages today, I noticed that installation of getSpatialData fails due to deprecation of devtools::system_check

Error: object ‘system_check’ is not exported by 'namespace:devtools'
Execution halted
ERROR: lazy loading failed for package ‘getSpatialData’
* removing ‘/home/lb/R/x86_64-pc-linux-gnu-library/3.6/getSpatialData’
* restoring previous ‘/home/lb/R/x86_64-pc-linux-gnu-library/3.6/getSpatialData’
Error: Failed to install 'testbf' from GitHub:
  Failed to install 'getSpatialData' from GitHub:
  (converted from warning) installation of package ‘/tmp/RtmpSdO1QS/file1bb20b7eddd/getSpatialData_0.0.4.tar.gz’ had non-zero exit status

(see https://github.com/r-lib/devtools/blob/c76ed73744a5ce9dd0bb824e63f5444c4f32a370/NEWS.md)

The function should be probably replaced by a call to

processx::process$new

I will draft a PR later on today (though AFAICS the function is never used within the package, so it seems it could be safely removed)

I tried running the "Parallelized download of MODIS data" example and when I ran this line:
records <- getMODIS_query(time_range = time_range, name = grep("MOD09GA", product_names, value = T))

I got this error message: Error in isFALSE(quiet) : could not find function "isFALSE"

I also tried records <- getMODIS_query(time_range = time_range, name = "MODIS_MYD14A1_V6")

and I got the same error message.

Is there is any provisional to download only specific bands from sentinel 2 such as B05 , B08

Actually i have work in these given bands for getting those i have to download all the data which is a waste of my bandwidth.

Api in Python has the eligibility to download only the required bands.

Hi,
I'm trying to use login_USGS on RStudio Server but the console show me the next error:
'Error: Login failed. Please retry later or call services_avail() to check if USGS services are currently unavailable.'
Is this an error from the USGS server or login_USGS not working in this version of Ubuntu?

All the best and thanks,
Matias

I'm trying to install the package on Debian 10:

> sessionInfo()
R version 3.5.2 (2018-12-20)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux 10 (buster)

Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/atlas/libblas.so.3.10.3
LAPACK: /usr/lib/x86_64-linux-gnu/atlas/liblapack.so.3.10.3

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_IL.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_IL.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_IL.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

and I then get many lines with errors when the installation tries to load the package:

> devtools::install_github("16EAGLE/getSpatialData")
Downloading GitHub repo 16EAGLE/getSpatialData@master
✔  checking for file ‘/tmp/RtmpyjMxMl/remotes662a762d7549/16EAGLE-getSpatialData-2525bf2/DESCRIPTION’ ...
─  preparing ‘getSpatialData’:
✔  checking DESCRIPTION meta-information ...
─  checking for LF line-endings in source and make files and shell scripts
─  checking for empty or unneeded directories
─  looking to see if a ‘data/datalist’ file should be added
─  building ‘getSpatialData_0.0.4.tar.gz’
   
Installing package into ‘/home/micha/R/x86_64-pc-linux-gnu-library/3.5’
(as ‘lib’ is unspecified)
* installing *source* package ‘getSpatialData’ ...
** R
** data
*** moving datasets to lazyload DB
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
Error relocating /usr/lib/x86_64-linux-gnu/libsqlite3.so.0: __memcpy_chk: symbol not found
Error relocating /usr/lib/x86_64-linux-gnu/libsqlite3.so.0: __memset_chk: symbol not found
Error relocating /usr/lib/x86_64-linux-gnu/libsqlite3.so.0: fcntl64: symbol not found
Error relocating /usr/lib/x86_64-linux-gnu/libexpat.so.1: __fprintf_chk: symbol not found
Error relocating /usr/lib/x86_64-linux-gnu/libexpat.so.1: getrandom: symbol not found
Error relocating /usr/lib/x86_64-linux-gnu/libopenjp2.so.7: __vsnprintf_chk: symbol not found
Error relocating /usr/lib/x86_64-linux-gnu/libopenjp2.so.7: __fprintf_chk: symbol not found
Error relocating /usr/lib/x86_64-linux-gnu/libopenjp2.so.7: __sprintf_chk: symbol not found
Error relocating /usr/lib/x86_64-linux-gnu/libgeotiff.so.2: __strcat_chk: symbol not found
Error relocating /usr/lib/x86_64-linux-gnu/libgeotiff.so.2: __vfprintf_chk: symbol not found
Error relocating /usr/lib/x86_64-linux-gnu/libgeotiff.so.2: __fprintf_chk: symbol not found
Error relocating /usr/lib/x86_64-linux-gnu/libgeotiff.so.2: __vsprintf_chk: symbol not found
Error relocating /usr/lib/x86_64-linux-gnu/libgeotiff.so.2: __sprintf_chk: symbol not found

Any suggestions how to overcome this?

Hello, and congrats for the great package. Im having a problem when setting the extent. More specifically, I want to download time-series data of Landsat 5 and 8 for the years 1987 until 2017. The study region is Greece (path-row). Even though I am setting the extent (show them the particular code), when the download process is starting it takes scenes only from the west part of the study area (specify the path-row of the scenes). Any thoughts? Here is the code:

devtools::install_github("16EAGLE/getSpatialData")

Load packages

library(getSpatialData)
library(raster)
library(sf)
library(sp)

#instead of using an existing AOI, you can simply draw one:
set_aoi() #call set_aoi() without argument, which opens a mapedit editor:

set AOI for this session

view_aoi() #view AOI in viewer, which will look like this:
aoi = get_aoi(type = "sf")

After defining a session AOI, define time range and platform

time_range <- c("2010-01-01", "2010-02-31")

set login credentials and archive directory

login_USGS(username = "vanto") #asks for password or define 'password'
set_archive("C:/Users/Vanto/Desktop/test")

Use getSentinel_query to search for data (using the session AOI)

records <- getLandsat_query(time_range = time_range, aoi = aoi)

Filter the records

colnames(records) #see all available filter attributes
unique(records$levels_available) #use one of the, e.g. to see available processing levels
unique(records$product)

records_filtered <- records[which(records$levels_available == "'source_metadata', 'l1', 'toa', 'bt', 'sr', 'sr_ndvi', 'sr_evi', 'sr_savi', 'sr_msavi', 'sr_ndmi', 'sr_nbr', 'sr_nbr2', 'stats', 'pixel_qa'"),] #filter by Level
records_filtered <- records[which(records$product == "LANDSAT_TM_C1"),] #filter by Level

records_filtered <- records_filtered[as.numeric(records_filtered$SceneCloudCover) <= 10, ] #filter by clouds

View records table

View(records)
View(records_filtered)
#browser records or your filtered records
getLandsat_preview(record = records_filtered[6,])

Dear all,
I am trying to download the Sentinel-5P NO2 product. Download suddenly stops in the middle of the download throwing out the following error.

library(getSpatialData)
library(raster)
library(sf)
library(sp)

## Define an AOI (either matrix, sf or sp object)
aoi=matrix(c(-10,60,-10,34,40,34,40,60),
           ncol = 2,byrow = TRUE)

## set AOI for this session
set_aoi(aoi)
view_aoi() #view AOI in viewer, which will look like this:
# #instead of using an existing AOI, you can simply draw one:
# set_aoi() #call set_aoi() without argument, which opens a mapedit editor:

login_CopHub(username = "**********",
             password = "*********")

set_archive("/path/")

records <- getSentinel_query(time_range = c("2020-02-01",
                                            "2020-02-30"), 
                             platform = "Sentinel-5P")
records_filtered <- records[records$producttype == "L2__NO2___",]

datasets <- getSentinel_data(records = records_filtered[1,])
  |=============================                                         |  

42%Error: Could not process request: transfer closed with 270422286 bytes remaining to read

Any help would be highly appreciated.
Best regards,
Omid.

Dear Mr,

First, many thanks for your work on this package. I used it for S-2 and S-1 and it's work well.

Then, i have to download sentinel images from the web site "http://www.theia-land.fr/" for my work.
So my question is, is it possible to download sentinel images from another site, not Copernicus but theia and can i do it?

Thanks for your answer.

Best regard,

C. ROZO

prepSentinel() produces a nested list of filenames with the specified extension (in my case, I tried both "vrt" and "tiff"), but the files in the list are nowhere to be found. Instead, the file paths in the list points to a set of text files with the same names as the filenames. Inside the text files are the names of the .jp2 that (presumably) the raster stack should be built from. I didn't think this was the intended behavior of prepSentinel(). Running R 3.5.2 and the most recent build of getSpatialData - downloaded 2 days ago from Github.

platform x86_64-w64-mingw32
arch x86_64
os mingw32
system x86_64, mingw32
status
major 3
minor 5.2
year 2018
month 12
day 20
svn rev 75870
language R
version.string R version 3.5.2 (2018-12-20)
nickname Eggshell Igloo

UPDATE: I cannot reproduce this issue, so I believe it was tied to an incorrect "solution" I had tried for the long nested filenames problem.

Data download via FTP no longer possible due to new NASA policies: LAADS deprecation of FTP access https://ladsweb.modaps.eosdis.nasa.gov/alerts-and-issues/?id=22211.

When I use orderrecord=order_data(records) for sentinel, I get an error as below,
Error: A column of 'records' named 'level' is required for this action, but is missing.

Currently the records is derived from the function get_records.
Since there is little information about the function order_data, I have no idea about the 'level' column, would you please tell me records should be exported from which function so that there will be a column of 'level'?

Many thanks.

Hi,

I have a problem when I run getLandsat_query in R. I get the next lines after run it:

Searching USGS EarthExplorer for available products...
Reading meta data of search results from USGS EarthExplorer...
Recieving available product levels from USGS-EROS ESPA...
Error in gSD.get(url = paste0(getOption("gSD.api")$espa, "available-products/", :
Moved Permanently (HTTP 301). Failed to connect to server.._

I saw that USGS-EROS ESPA server is on maintenance, is that the answer of this error?

Greetings and thank you!

Hi, I'm having trouble downloading Landsat data. I've used this package for Sentinel 2 and love it! I successfully log in using my USGS Account and getLandsat_records.

records_Landsat <- getLandsat_records(time_range = c(start_records, end_records), products = "LANDSAT_8_C1")

However, when I try to use these records to access a dataset,

datasets_Landsat <- getLandsat_data(records = records_Landsat_filtered)

I get this error:

Error in .get(paste0(getOption("gSD.api")$espa, "/order/", x), getOption("gSD.usgs_user"), : Internal Server Error (HTTP 500). Failed to process request.

It seems like a strange error so I'm likely doing something wrong... but I can't figure it out.

This is mostly a reminder to myself to submit a PR.

Ideas here: http://ncss-tech.github.io/AQP/soilDB/SDA-tutorial-2.html

Congrats for this package which is really useful. For the users of Docker, i have built the Docker repo atavares/rocker-geospatial-getspatialdata. The repo is is automatically updated with every new build of rocker/geospatial meaning that is kept updated with the latest R, R geospatial packages, R Studio server and dev version of getSpatialData (this github repo).

Fantastic initiative!

I writed codes to download EcoClimate, CHELSA, WorldClim v1.4 and v2.0, SoilGrids and some others.

If have interest, please send me an e-mail: [email protected]

I will happy in contribute.

Congrats again!

Sen2Cor is a processor for Sentinel-2 Level 2A product generation and formatting; it performs the atmospheric-, terrain and cirrus correction of Top-Of- Atmosphere Level 1C input data. Sen2Cor creates Bottom-Of-Atmosphere, optionally terrain- and cirrus corrected reflectance images; additional, Aerosol Optical Thickness-, Water Vapor-, Scene Classification Maps and Quality Indicators for cloud and snow probabilities. Its output product format is equivalent to the Level 1C User Product: JPEG 2000 images, three different resolutions, 60, 20 and 10 m.

Sec2Cor ESA webpage:
http://step.esa.int/main/third-party-plugins-2/sen2cor/

Sen2Cor on Docker:
https://hub.docker.com/r/atavares/sen2cor/
https://bitbucket.org/atntavares/sen2cor/src/master/

I am not able to download preview images for Sentinel-2 products. I am however able to download Landsat-8 previews and if I go to the link in the preview_url column I am able to see a Sentinel-2 preview. Furthermore, verifying availability of data and ordering data is no problem.

library(getSpatialData)

#login
login_USGS(username = "myuser")

# session settings
set_archive(dir_archive = "path/to/archive")
data("aoi_data")
# Define an area of interest (AOI):
# Use the example AOI or draw an AOI by calling set_aoi():
set_aoi(aoi_data[[1]])
# View the AOI:
view_aoi()

login_USGS(username = "", password = "")
login_CopHub(username = "", password = "")

# session settings
set_archive("./pictures2/")
set_aoi(aoi_data[[1]])

# query
records <- get_records(time_range = c("2018-05-15", "2018-05-22"),
                       products = c("LANDSAT_8_C1", "Sentinel-2"))




records_landsat <- records[records$level == "sr",] # filter level 1 records
records_landsat <- get_previews(records_landsat)
view_previews(records_landsat[1,])

Downloading previews...
[Preview 1/4] Skipping download of preview 'LC08_L1TP_195025_20180518_20180604_01_T1', since './pictures2/_previews/LANDSAT_8_C1/LC08_L1TP_195025_20180518_20180604_01_T1_preview.jpg' already exists...
[Preview 2/4] Skipping download of preview 'LC08_L1TP_195026_20180518_20180604_01_T1', since './pictures2/_previews/LANDSAT_8_C1/LC08_L1TP_195026_20180518_20180604_01_T1_preview.jpg' already exists...
[Preview 3/4] Skipping download of preview 'LC08_L1TP_193025_20180520_20180605_01_T1', since './pictures2/_previews/LANDSAT_8_C1/LC08_L1TP_193025_20180520_20180605_01_T1_preview.jpg' already exists...
[Preview 4/4] Skipping download of preview 'LC08_L1TP_193026_20180520_20180605_01_T1', since './pictures2/_previews/LANDSAT_8_C1/LC08_L1TP_193026_20180520_20180605_01_T1_preview.jpg' already exists...

Georeferncing previews...
[Preview 1/4] Skipping converting of './pictures2/_previews/LANDSAT_8_C1/LC08_L1TP_195025_20180518_20180604_01_T1_preview.jpg', since './pictures2/_previews/LANDSAT_8_C1/LC08_L1TP_195025_20180518_20180604_01_T1_preview.tif' already exists...
[Preview 2/4] Skipping converting of './pictures2/_previews/LANDSAT_8_C1/LC08_L1TP_195026_20180518_20180604_01_T1_preview.jpg', since './pictures2/_previews/LANDSAT_8_C1/LC08_L1TP_195026_20180518_20180604_01_T1_preview.tif' already exists...
[Preview 3/4] Skipping converting of './pictures2/_previews/LANDSAT_8_C1/LC08_L1TP_193025_20180520_20180605_01_T1_preview.jpg', since './pictures2/_previews/LANDSAT_8_C1/LC08_L1TP_193025_20180520_20180605_01_T1_preview.tif' already exists...
[Preview 4/4] Skipping converting of './pictures2/_previews/LANDSAT_8_C1/LC08_L1TP_193026_20180520_20180605_01_T1_preview.jpg', since './pictures2/_previews/LANDSAT_8_C1/LC08_L1TP_193026_20180520_20180605_01_T1_preview.tif' already exists...
Columns added to records: 'preview_file_jpg', 'preview_file'

However, this does not work

records_sentinel <- records[records$level == "Level-2A",] # filter level 1 records
records_sentinel <- get_previews(records_sentinel)

Downloading previews...
[Preview 1/12] Downloading preview 'S2B_MSIL2A_20180520T103019_N0207_R108_T32UNA_20180520T124443' to './pictures2/_previews/Sentinel-2/S2B_MSIL2A_20180520T103019_N0207_R108_T32UNA_20180520T124443_preview.jpg'...
Warning: [Preview 1/12] Download of preview 'S2B_MSIL2A_20180520T103019_N0207_R108_T32UNA_20180520T124443' failed: Unauthorized (HTTP 401). Failed to process request.
[Preview 2/12] Downloading preview 'S2B_MSIL2A_20180520T103019_N0207_R108_T32UNV_20180520T124443' to './pictures2/_previews/Sentinel-2/S2B_MSIL2A_20180520T103019_N0207_R108_T32UNV_20180520T124443_preview.jpg'...
Warning: [Preview 2/12] Download of preview 'S2B_MSIL2A_20180520T103019_N0207_R108_T32UNV_20180520T124443' failed: Unauthorized (HTTP 401). Failed to process request.
[Preview 3/12] Downloading preview 'S2B_MSIL2A_20180520T103019_N0207_R108_T32UPV_20180520T124443' to './pictures2/_previews/Sentinel-2/S2B_MSIL2A_20180520T103019_N0207_R108_T32UPV_20180520T124443_preview.jpg'...
Warning: [Preview 3/12] Download of preview 'S2B_MSIL2A_20180520T103019_N0207_R108_T32UPV_20180520T124443' failed: Unauthorized (HTTP 401). Failed to process request.
[Preview 4/12] Downloading preview 'S2B_MSIL2A_20180520T103019_N0207_R108_T32UPA_20180520T124443' to './pictures2/_previews/Sentinel-2/S2B_MSIL2A_20180520T103019_N0207_R108_T32UPA_20180520T124443_preview.jpg'...
Warning: [Preview 4/12] Download of preview 'S2B_MSIL2A_20180520T103019_N0207_R108_T32UPA_20180520T124443' failed: Unauthorized (HTTP 401). Failed to process request.

If I try to actually download the full data, there is no problem

records_sentinel <- order_data(records_sentinel)
records_sentinel <- get_data(records_sentinel)

Code is running all ok .
When i start download using command line datasets <- getSentinel_data(records = records_filtered[1,])

Download starts but its stops at any percentage such as 65%,17%,25% but never completed to 100%.Earlier it used to be completed to 100% but now it seems to fail.Tried network change still no solution

Any solution is appreciated..

# Get Sentinel-3 Data

packages_demanded <- c("sp", "rgdal", "raster", "rgeos", "RStoolbox", "devtools", "water")
install.packages(packages_demanded)
lapply(packages_demanded, require, character.only = TRUE)

devtools::install_github("16eagle/getSpatialData")
library(getSpatialData)

setwd("C:/Users/UnknownUser")
wd <- getwd()

epsg4326 <- "+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs"
aoi_epsg4326 <- createAoi(topleft = c(-9.136454, 41.138178), bottomright = c(-7.310318, 38.707536), EPSG = 4326)
aoi_epsg4326 <- as(aoi_epsg4326, "SpatialPolygonsDataFrame")
# create AOI

login_CopHub("UnknownUser")

set_aoi(aoi_epsg4326)

query <- getSentinel_query(time_range = c("2017-06-17", "2017-06-19"), platform = "Sentinel-3")

query_filt_1 <- query[query$producttype == "SL_1_RBT___", ]
query_filt_2 <- query_filt_1[7:8, ]

getSentinel_data(records = query_filt_2, dir_out = paste0(wd, "/Data/Sentinel3"))
# results in following ERROR
# Error in !records$online : invalid argument type

I don't understand how to deal with sentinel 5p .nc file format.
prepSentinel seems to expect a zip file, but the .nc is not.
How to actually load .nc files?
Thank you

See issue#10, copied below for convenience:

Maybe some improvements to add to prepSentinel() are:

• add somehow the names of the bands to the output .tif files. The L2A outputs are available with the names below in the original .SAFE data, but with prepSentinel() the band names get lost:

band.order <- list("aux" = c("AOT", "CLD", "SCL", "TCI", "WVP", "SNW"), "refl" = c("B01", "B02", "B03", "B04", "B05", "B06", "B07", "B08", "B8A", "B09", "B10", "B11", "B12"))

• add a function input variable SAFE = true to avoid step 4 above (create a .zip file)

Hi,
I can't load the getSpatialData anymore because of a gdal error:

the error message is:

probably broken install of gdal at 'C:/Qgis/apps/gdal2/bin/' Error: package or namespace load failed for ‘getSpatialData’: .onLoad failed in loadNamespace() for 'getSpatialData', details: call: strsplit(sapply(drivers, function(x) { error: non-character argument In addition: Warning messages: 1: In FUN(X[[i]], ...) : running command '"C:\Qgis\apps\gdal2\bin\gdalinfo.exe" --version' had status 309 2: In FUN(X[[i]], ...) : running command '"C:/Qgis/apps/gdal2/bin//gdalinfo" --formats' had status 309

anybody has an idea how to fix this?

Thanks !

When using getMODIS_preview the preview seems like it might not be projected appropriately. For example, when using getMODIS_preview as in Issue #22 this is how it displays:

However - when downloading the data separately, using rgdal/raster to import the data, and mapview to view the data, this is the result (and land cover types make sense when using an imagery-based baselayer):

Thus, it just seems like something is off when using the preview image - it looks like the getMODIS_preview function, the image actually gets stretched to the bounding box of the MODIS tile.

When I run the below code from this repository I get the following error. Any insight would be appreciated.

the error occurs when view_aoi() is run.

Error in htmlwidgets::sizingPolicy(defaultWidth = defaultWidth, defaultHeight = defaultHeight, :
unused argument (browser.external = FALSE)

Load packages

library(getSpatialData)
library(raster)
library(sf)
library(sp)

Define an AOI (either matrix, sf or sp object)

data("aoi_data") # example aoi

aoi <- aoi_data[[3]] # AOI as matrix object, or better:
aoi <- aoi_data[[2]] # AOI as sp object, or:
aoi <- aoi_data[[1]] # AOI as sf object
#instead, you could define an AOI yourself, e.g. as simple matrix

Define an AOI (either matrix, sf or sp object)

data("aoi_data") # example aoi

aoi <- aoi_data[[3]] # AOI as matrix object, or better:
aoi <- aoi_data[[2]] # AOI as sp object, or:
aoi <- aoi_data[[1]] # AOI as sf object
#instead, you could define an AOI yourself, e.g. as simple matrix

set AOI for this session

set_aoi(aoi)
view_aoi() #view AOI in viewer, which will look like this:

set_aoi()

Hello,
I am trying to use the function getSentinel_query exactly following the example at the main page with only difference that I am using Sentinel 3 platform.
I am receiving the following error message:

Error in doc_parse_raw(x, encoding = encoding, base_url = base_url, as_html = as_html,  : 
  xmlParseEntityRef: no name [68]
In addition: Warning messages:
1: In doc_parse_raw(x, encoding = encoding, base_url = base_url, as_html = as_html,  :
  Entity 'nbsp' not defined [27]
2: In doc_parse_raw(x, encoding = encoding, base_url = base_url, as_html = as_html,  :
  Entity 'nbsp' not defined [27]

It seems that the problem arises in

as_xml_document(content(query, 
            as = "text"))

which is in getSentinel_query.R code. If ' ' is replaced by ' ' using gsub function in this code, the warning disappears, however, the error in doc_parse_raw remains.

I am using getSpatialData_0.0.4 with R 3.6.0 on Windows 10.

Please, do you have any suggestions?

Best,
Milan