add index about modis-ingestor HOT 7 OPEN

looking for some feedback on what should be included in this index @jflasher @drewbo @scisco
So far, definitely
granuleid
acquisitiondate - it's in the filename, but worth having as a separate field
download_url (for landsat-pds this is the https address so points to the index.html file)

I'm not convinced that bounding box coords are needed, but am open to including them if there's strong enough feeling about it. I think it's a lot of redundant info since the tile x, y coordinates define the bbox.

cloudcover is not pertinent, processinglevel for MODIS is essentially defined by the product name (e.g., MCD43A4), but that's included in the granule ID (e.g., MCD43A4.A2017006.h01v08.006.2017018075239 )

I'd also recommend tossing in a shapefile or geojson file in the top level (in s3://modis-pds/) of the MODIS tile grid, possibly in both the original sinusoidal and a cleaned up geographic (including both is helpful because users who transform the sin grid to geographic will find a pretty wonky looking grid due to the prime meridian issue).

from modis-ingestor.

Agreed. If we have the image coordinates in the metadata, bounding box is redundant. Having cloudCover in the metadata saves time. Also love the idea of having a top level geojson.

from modis-ingestor.

There was a request for night/day for Landsat as well, but don't think that applies here? For the coordinates, I think it'd be nice if someone can use the index file to find all the scenes that contain a lat/lon. I think this is easiest if the bbox is in the index file, but if someone can get the matching x,y from a top level geojson and then search for those x, y matches in the index file, I think that could work as well?

from modis-ingestor.

night/day doesn't apply here, only to the Land Surface Temp products, in which case the daytime and nighttime temps are actually included in the same product (MOD11A1 and MYD11A1) as separate bands.

I think it's better to find the tileid for a geo request, then find matching tile ids, rather than doing a search on tens of thousands of tiles when there is only about 300 unique ones.

Additionally, since MCD43 is a composite over 16 days that is (theoretically) cloud free, a cloud cover % isn't available, nor does it apply.

from modis-ingestor.

👍

from modis-ingestor.

index files for each date are added under the the product name:
e.g.,
s3://modis-pds/MCD43A4.006/2017-02-10_scenes.txt

This is because a day is processed in a batch. If the entire day is processed then the the index file for that day is uploaded. This avoids complications of trying to write the same file without there being an index for every granule. If the process is interrupted then no day index gets uploaded and the whole day will be reprocessed during normal gap-filling*.

Example of a day scene index looks like this:

date,download_url,gid
2017-01-07 00:00:00,https://modis-pds.s3.amazonaws.com/MCD43A4.006/23/01/2017007/index.html,MCD43A4.A2017007.h23v01.006.2017018073630
2017-01-07 00:00:00,https://modis-pds.s3.amazonaws.com/MCD43A4.006/22/01/2017007/index.html,MCD43A4.A2017007.h22v01.006.2017018073712
2017-01-07 00:00:00,https://modis-pds.s3.amazonaws.com/MCD43A4.006/18/09/2017007/index.html,MCD43A4.A2017007.h18v09.006.2017018075910

As we talked about however a single scene index is desirable, so each day all the available daily scene index files are concatenated into a single scene.txt which is uploaded to the product "folder" (e.g., s3://modis-pds/MCD43A4.006)**

*I've not yet added a cronjob to do this, but this adds redundancy by running every few days and checking if all days are accounted for.

** Also have not yet added the job to do this, but there are some scripts available to do this efficiently (e.g., https://gist.github.com/jasonrdsouza/f2c77dedb8d80faebcf9)

Will leave this ticket open until the scene.txt file is added.

cc @jflasher @drewbo

from modis-ingestor.

😻

from modis-ingestor.