Giter VIP home page Giter VIP logo

Comments (6)

16EAGLE avatar 16EAGLE commented on June 7, 2024

Thanks, I would like to look into what goes wrong here, however, your example fails with

Rects$BR_Lat = Locations$Lat
#> Error in Rects$BR_Lat = Locations$Lat : object 'Rects' not found

since Rects is not defined. Could you please update your example, so that I can run the exact query that you ran?

from getspatialdata.

fishish1 avatar fishish1 commented on June 7, 2024

Sorry about that. Here's a small change to it:

Locations = data.frame(39.9,-105.65)
colnames(Locations)=c("Lat","Lon")

cell_dim = .1
Rects=Locations
Rects$BR_Lat = Locations$Lat
Rects$BR_Lon = Locations$Lon
Rects$TR_Lat = Rects$BR_Lat+cell_dim
Rects$TR_Lon= Rects$BR_Lon
Rects$TL_Lat= Rects$TR_Lat
Rects$TL_Lon= Rects$TR_Lon - cell_dim
Rects$BL_Lat= Rects$BR_Lat
Rects$BL_Lon=Rects$TL_Lon
Rects=Rects[,3:10]

a<-matrix(as.numeric(as.character(Rects[1, ])), ncol=2, byrow=TRUE)

set_aoi(a)
view_aoi() #view AOI in viewer, which will look like this:

start_records = "2020-06-01"
end_records = "2020-07-05"

#records = get_records(time_range = c(start_records, end_records), products = c("LANDSAT_8_C1", "Sentinel-2", "LANDSAT_ETM_C1"))
records = get_records(time_range = c(start_records, end_records), products = c("LANDSAT_8_C1"))

records_filtered <- records[which(records$level %in% c("l1")),] #filter by Level
records_filtered <- records_filtered[as.numeric(records_filtered$cloudcov) <= 70, ] #filter by clouds

#order any data that is not already available 
test_for_availability = check_availability(records_filtered)

tryCatch(records_filtered <- order_data(test_for_availability), error = function(e) message("No New Records Ordered: ", e))

#test for all data availability for 60 minutes else proceed with current data
time_waited=0
cycle=60

#try for an hour before just going with all avaialble now
while(!all(test_for_availability$download_available==TRUE && time_waited<3600)){
  Sys.sleep(cycle)
  test_for_availability = check_availability(records_filtered)
  time_waited=time_waited+cycle
}

records_filtered = test_for_availability[test_for_availability$download_available==TRUE,]
records_filtered <-records_filtered[order(records_filtered$cloudcov),]

get_data(records = records_filtered[1,])
get_data(records = records_filtered[2,])

from getspatialdata.

16EAGLE avatar 16EAGLE commented on June 7, 2024

Thanks. When I run your code, view_aoi() on your AOI results in

#> Warning message:
#> In sf::st_is_longlat(x) :
#>   bounding box has potentially an invalid value range for longlat data

The reason appears to be that longitude and latitude have been confused in a<-matrix(as.numeric(as.character(Rects[1, ])), ncol=2, byrow=TRUE). When defining the AOI as a matrix, set_aoi expects the column order to be longitude, latitude (when I run your code, a has the opposite column order). I am not sure how you were able to query for records with that AOI since this resulted in zero records for me.

This however works fine for me and I do not get a 404

library(getSpatialData)
library(sf)

# using your corner coordinates to define AOI with column order (lon, lat)
aoi <- cbind(c(-105.65, -105.65, -105.75, -105.75), #longitude
             c(39.9, 40.0, 40.0, 39.9)) # latitude
aoi <- rbind(aoi, aoi[1,]) # close the polygon
aoi <- st_sfc(st_polygon(list(aoi)), crs = 4326)

set_aoi(aoi)
view_aoi()
set_archive("archive/")

start_records = "2020-06-01"
end_records = "2020-07-05"

login_USGS("user")
records = get_records(time_range = c(start_records, end_records), products = c("LANDSAT_8_C1"))

records <- records[which(records$level %in% c("l1")),] #filter by Level
records <- records[as.numeric(records$cloudcov) <= 70, ] #filter by clouds

#order any data that is not already available 
records <- check_availability(records)
get_data(records)
#> Receiving MD5 checksums...
#> Assembling dataset URLs...
#> [Dataset 1/2 | File 1/38] Download dataset 'LC08_L1TP_034032_20200616_20200626_01_T1 (l1)' to
#> '/home/xxxx/xxxxx/tmp/gSD//LANDSAT_8_C1//LC08_L1TP_034032_20200616_20200626_01_T1/LC08_L1TP_034032_20200616_20200626_01_T1_B6.TIF'...
#>   |=====================================================================================| 100%

Side note: I see you build a little while loop to check for orders to be ready for download. You can also simply use the argument wait_to_complete = TRUE in order_data() to make the function run until the last order is complete.

from getspatialdata.

fishish1 avatar fishish1 commented on June 7, 2024

I used your code and verified I successfully get the data. However, I think this is an issue that has to do with timing.

I changed the aoi and timing slightly to get the most recent Landsat data near me. One I did this I received the same 404:

library(getSpatialData)
library(sf)

# using your corner coordinates to define AOI with column order (lon, lat)
aoi <- cbind(c(-115.65, -115.65, -105.75, -105.75), #longitude
             c(39.9, 45.0, 45.0, 39.9)) # latitude
aoi <- rbind(aoi, aoi[1,]) # close the polygon
aoi <- st_sfc(st_polygon(list(aoi)), crs = 4326)

set_aoi(aoi)
view_aoi()
set_archive("archive/")

start_records = "2020-07-13"
end_records = "2020-07-14"

login_USGS("user")
records = get_records(time_range = c(start_records, end_records), products = c("LANDSAT_8_C1"))

records <- records[which(records$level %in% c("l1")),] #filter by Level
records <- records[as.numeric(records$cloudcov) <= 70, ] #filter by clouds

#order any data that is not already available 
records <- check_availability(records)
get_data(records)

##>Error in .get(index_url) : 
 ##>Not Found (HTTP 404). Failed to process request.

from getspatialdata.

fishish1 avatar fishish1 commented on June 7, 2024

I've discovered the root cause of this issue. There is a mismatch between the products which are available from USGS versus from AWS. When using awscli, I can verify that products (from 2020-08-03), which are available through NASA Earth Explorer, are not yet available on the aws pds.

AWS states: "New Landsat 8 scenes are added regularly as soon as they are available."

Clearly, this is not very truthful wording from Amazon.

from getspatialdata.

fishish1 avatar fishish1 commented on June 7, 2024

I'll close this issue for now because I don't wish the task of interpreting the Earth Explorer API documentation upon anyone. However, I think the right solution to most problems is avoiding AWS.

from getspatialdata.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.