opengeos / maxar-open-data Goto Github PK

View Code? Open in Web Editor NEW

198.0 7.0 16.0 20.23 MB

The Maxar Open Data STAC Catalog in CSV, GeoJSON, and MosaicJSON formats

License: MIT License

Python 100.00%

cog geospatial geotiff maxar satellite-imagery stac-catalog

maxar-open-data's People

Contributors

Stargazers

Watchers

Forkers

yputthealy lpavankumar davidchoi76 wanwanliang changwan-planet zsmilliepy curtis18 senyaofeng aabognah badermadich dcyd amaan-samar vrym2 barisdev drroad ashkanmosalla

maxar-open-data's Issues

Incorrect URL in streamlit_app.py

Dear Professor, thank you very much for all your contributions to the open geospatial world.

I have noticed that this streamlit_app.py is no longer working (https://maxar-open-data.streamlit.app/). Probably because the URL "https://giswqs.github.io/maxar-open-data" no longer exists. I've noticed that the repo (https://github.com/giswqs/maxar-open-data/blob/master/datasets ) also redirects to "https://github.com/opengeos/maxar-open-data/tree/master/datasets". Kindly look into this mater, as this application was found to be very useful to find pre and post disaster images during an emergency.
Thank you.

~60 datasets from 2010 - mid-2021?

There appears to be ~60 datasets listed on https://www.maxar.com/open-data which aren't linked to in this repo. Are there any plans to add them?

KeyError: 'tiles'

Hi, I am trying to reproduce the example to load and visualize Maxar images:

Leafmap version:
0.21.3

m = leafmap.Map()
url = 'https://open.gishub.org/maxar-open-data/datasets/Kahramanmaras-turkey-earthquake-23/1050050044DE7E00.json'
m.add_stac_layer(url, name="Mosaic")
m

But I am getting this error:

KeyError Traceback (most recent call last)
Cell In[11], line 3
1 m = leafmap.Map()
2 url = 'https://open.gishub.org/maxar-open-data/datasets/Kahramanmaras-turkey-earthquake-23/1050050044DE7E00.json'
----> 3 m.add_stac_layer(url, name="Mosaic")
4 m

File ~\miniconda3\envs\geo\lib\site-packages\leafmap\leafmap.py:924, in Map.add_stac_layer(self, url, collection, item, assets, bands, titiler_endpoint, name, attribution, opacity, shown, fit_bounds, **kwargs)
894 def add_stac_layer(
895 self,
896 url=None,
(...)
907 **kwargs,
908 ):
909 """Adds a STAC TileLayer to the map.
910
911 Args:
(...)
922 fit_bounds (bool, optional): A flag indicating whether the map should be zoomed to the layer extent. Defaults to True.
923 """
--> 924 tile_url = stac_tile(
925 url, collection, item, assets, bands, titiler_endpoint, **kwargs
926 )
927 bounds = stac_bounds(url, collection, item, titiler_endpoint)
928 self.add_tile_layer(tile_url, name, attribution, opacity, shown)

File ~\miniconda3\envs\geo\lib\site-packages\leafmap\stac.py:674, in stac_tile(url, collection, item, assets, bands, titiler_endpoint, **kwargs)
671 else:
672 r = requests.get(titiler_endpoint.url_for_stac_item(), params=kwargs).json()
--> 674 return r["tiles"][0]

KeyError: 'tiles'

Thanks in advance for looking into it.

Visualizing the Morrocco "after" datasets

Hi there and thanks for the work!

I am trying to visualize the Morrocco dataset and followed the examples on how to get the collection of the "after" pictures.
However I can't figure out how to use the collection to actually visualize the iamges themseves.

Any hint would be much apreciated.

ValueError: '10300500E4F91900' is not in list

When we change the dataset, it can not find previously selected COG mosaic and throws below error:

ValueError: '10300500E4F91900' is not in list
Traceback:
File "/home/user/.local/lib/python3.9/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 565, in _run_script
exec(code, module.dict)
File "/home/user/app/app.py", line 41, in
catalog = st.selectbox('Select a COG mosaic', get_catalogs(dataset), index=get_catalogs(dataset).index('10300500E4F91900'))

Any plans to include metadata URLs alongside the visual.tif URLs?

Maxar's ARD deliverables can contain ~17 metadata files for any given visual.tif https://ard.maxar.com/docs/getting-started/about-sample-data-001/

📁 {bucket) maxar-data-samples
└── 📁 {prefix} sample-001
    └── 📁 {UTM-zone} 10
        └── 📁 {quadkey} 120020223032
            └── 📁 {date of acquisition} 2016-10-08
            └─╴📄 103001005D31F500-cloud-mask.gpkg
            └─╴📄 103001005D31F500-cloud-shadow-mask.gpkg
            └─╴📄 103001005D31F500-clouds.tif
            └─╴📄 103001005D31F500-data-mask.gpkg
            └─╴📄 103001005D31F500-healthy-vegetation.tif
            └─╴📄 103001005D31F500-healthy-vegetation-mask.gpkg
            └─╴📄 103001005D31F500-pan-flare.tif
            └─╴📄 103001005D31F500-pan-flare-mask.tif
            └─╴📄 103001005D31F500-ms-saturation-mask.gpkg
            └─╴📄 1103001005D31F500-ms-saturation.tif
            └─╴📄 103001005D31F500-ms.tif
            └─╴📄 103001005D31F500-pan.tif
            └─╴📄 103001005D31F500-terrain-shadow-mask.gpkg
            └─╴📄 103001005D31F500-terrain-shadows.tif
            └─╴📄 103001005D31F500-visual.tif
            └─╴📄 103001005D31F500-water-mask.gpkg
            └─╴📄 103001005D31F500-water.tif
            └─╴📄 103001005D31F500.json

I tested some URLs and they do look to occasionally be accessible for some of the datasets linked to in this project.

$ for FILENAME in 10300100DE4D9300-cloud-mask.gpkg \
                  10300100DE4D9300-cloud-shadow-mask.gpkg \
                  10300100DE4D9300-clouds.tif \
                  10300100DE4D9300-data-mask.gpkg \
                  10300100DE4D9300-healthy-vegetation.tif \
                  10300100DE4D9300-healthy-vegetation-mask.gpkg \
                  10300100DE4D9300-pan-flare.tif \
                  10300100DE4D9300-pan-flare-mask.tif \
                  10300100DE4D9300-ms-saturation-mask.gpkg \
                  10300100DE4D9300-ms.tif \
                  10300100DE4D9300-pan.tif \
                  10300100DE4D9300.json; do
      RESP=`curl -Is https://maxar-opendata.s3.amazonaws.com/events/New-Zealand-Flooding23/ard/60/213311212310/2022-12-11/$FILENAME | head -n1`
      if [[ $RESP == *"HTTP/1.1 200 OK"* ]]; then
          echo $FILENAME
      fi
  done

10300100DE4D9300-data-mask.gpkg
10300100DE4D9300-ms.tif
10300100DE4D9300-pan.tif
10300100DE4D9300.json

Any plans to name these URLs specifically within the GeoJSON manifest files?

How to get longitude/Latitudes as well as images for these events?

Hi, very excited about this repo. How can we retrieve the lon/lat for a tile?

New-Zealand-Flooding22 images have gone missing

The TIFFs linked to in https://raw.githubusercontent.com/opengeos/maxar-open-data/master/datasets/New-Zealand-Flooding22.geojson are offline.

This one for example returns the following XML:

https://maxar-opendata.s3.amazonaws.com/events/New-Zealand-Flooding22/ard/60/213311212301/2022-12-11/10300100DE4D9300-visual.tif

<Error>
<Code>NoSuchKey</Code>
<Message>The specified key does not exist.</Message>
<Key>events/New-Zealand-Flooding22/ard/60/213311212301/2022-12-11/10300100DE4D9300-visual.tif</Key>
<RequestId>PBMD5ZV4QDVHAC2T</RequestId>
<HostId>jMkQBeojPAnMZjzLg1LNorZgnykROPgfklr6JBWjW1HgZ2V/qoUgVXaPt+yhpZnW+bBjUBRz2BI=</HostId>
</Error>

TIFF Stack Explanations?

Any idea what the 14 layers within any one TIFF represent?

$ tiffinfo ard_60_213311212330_2023-02-08_10300100E23B4300-visual.tif

TIFF Directory at offset 0x8 (8)
  Image Width: 17408 Image Length: 17408
  Tile Width: 512 Tile Length: 512
  Bits/Sample: 8
  Sample Format: unsigned integer
  Compression Scheme: JPEG
  Photometric Interpretation: YCbCr
  YCbCr Subsampling: 2, 2
  Samples/Pixel: 3
  Planar Configuration: single image plane
  Reference Black/White:
     0:     0   255
     1:   128   255
     2:   128   255
  Tag 33550: 0.305176,0.305176,0.000000
  Tag 33922: 0.000000,0.000000,0.000000,289843.750000,5930156.250000,0.000000
  Tag 34735: 1,1,0,7,1024,0,1,1,1025,0,1,1,1026,34737,22,0,2049,34737,7,22,2054,0,1,9102,3072,0,1,32760,3076,0,1,9001
  Tag 34737: WGS 84 / UTM zone 60S|WGS 84|
  JPEG Tables: (142 bytes)
TIFF Directory at offset 0x2682 (9858)
  Subfile Type: transparency mask (4 = 0x4)
  Image Width: 17408 Image Length: 17408
  Tile Width: 512 Tile Length: 512
  Bits/Sample: 1
  Sample Format: unsigned integer
  Compression Scheme: AdobeDeflate
  Photometric Interpretation: transparency mask
  Samples/Pixel: 1
  Planar Configuration: single image plane
  Predictor: none 1 (0x1)
...
  Sample Format: unsigned integer
  Compression Scheme: AdobeDeflate
  Photometric Interpretation: transparency mask
  Samples/Pixel: 1
  Planar Configuration: single image plane
  Predictor: none 1 (0x1)

$ convert ard_60_213311212330_2023-02-08_10300100E23B4300-visual.tif x%d.tiff
$ ls -lhS x*.tiff

Below I've noted the size of each TIFF and if they render in Windows Explorer or not.

Size	Filename	Renders in File Explorer?
868M	x0.tiff	Yes, renders fine
217M	x2.tiff	Yes, renders fine
55M	x3.tiff	Yes, renders fine
14M	x4.tiff	Yes, renders fine
3.4M	x5.tiff	Yes, renders fine
868K	x6.tiff	Yes, renders fine
218K	x7.tiff	Yes, renders fine
37K	x1.tiff	No, the image is completely black
9.3K	x8.tiff	No, the image is completely black
2.6K	x9.tiff	No, the image is completely black
854	x10.tiff	No, the image is completely black
424	x11.tiff	No, the image is completely black
318	x12.tiff	No, the image is completely black
290	x13.tiff	No, the image is completely black

This one doesn't render in windows explorer.

$ tiffinfo x1.tiff

TIFF Directory at offset 0x8fe4 (36836)
  Image Width: 17408 Image Length: 17408
  Bits/Sample: 1
  Compression Scheme: AdobeDeflate
  Photometric Interpretation: min-is-black
  FillOrder: msb-to-lsb
  Orientation: row 0 top, col 0 lhs
  Samples/Pixel: 1
  Rows/Strip: 17408
  Planar Configuration: single image plane
  Page Number: 1-14
  White Point: 0.3127-0.329
  PrimaryChromaticities: 0.640000,0.330000,0.300000,0.600000,0.150000,0.060000

When I drag it onto QGIS is renders a huge black box that starts at Null Island and renders off the edge of the earth.

I pulled some metadata from another file and loaded it into a DuckDB table. It gives me technical details but no explanation as to what each image represents.

┌───────────┬────────────────────┬─────────────┬────────────────────────────┬──────────────────────┬──────────────────┬───────────────────┬──────────────┬──────────────────────────────────────────────────────┐
│ stack_num │ Compression Scheme │ JPEG Tables │ Photometric Interpretation │ Planar Configuration │  Sample Format   │ YCbCr Subsampling │  Predictor   │                     Subfile Type                     │
│  uint64   │      varchar       │   varchar   │          varchar           │       varchar        │     varchar      │      varchar      │   varchar    │                       varchar                        │
├───────────┼────────────────────┼─────────────┼────────────────────────────┼──────────────────────┼──────────────────┼───────────────────┼──────────────┼──────────────────────────────────────────────────────┤
│         1 │ JPEG               │ (142 bytes) │ YCbCr                      │ single image plane   │ unsigned integer │ 2, 2              │              │                                                      │
│         2 │ AdobeDeflate       │             │ transparency mask          │ single image plane   │ unsigned integer │                   │ none 1 (0x1) │ transparency mask (4 = 0x4)                          │
│         3 │ JPEG               │ (142 bytes) │ YCbCr                      │ single image plane   │ unsigned integer │ 2, 2              │              │ reduced-resolution image (1 = 0x1)                   │
│         4 │ JPEG               │ (142 bytes) │ YCbCr                      │ single image plane   │ unsigned integer │ 2, 2              │              │ reduced-resolution image (1 = 0x1)                   │
│         5 │ JPEG               │ (142 bytes) │ YCbCr                      │ single image plane   │ unsigned integer │ 2, 2              │              │ reduced-resolution image (1 = 0x1)                   │
│         6 │ JPEG               │ (142 bytes) │ YCbCr                      │ single image plane   │ unsigned integer │ 2, 2              │              │ reduced-resolution image (1 = 0x1)                   │
│         7 │ JPEG               │ (142 bytes) │ YCbCr                      │ single image plane   │ unsigned integer │ 2, 2              │              │ reduced-resolution image (1 = 0x1)                   │
│         8 │ JPEG               │ (142 bytes) │ YCbCr                      │ single image plane   │ unsigned integer │ 2, 2              │              │ reduced-resolution image (1 = 0x1)                   │
│         9 │ AdobeDeflate       │             │ transparency mask          │ single image plane   │ unsigned integer │                   │ none 1 (0x1) │ reduced-resolution image/transparency mask (5 = 0x5) │
│        10 │ AdobeDeflate       │             │ transparency mask          │ single image plane   │ unsigned integer │                   │ none 1 (0x1) │ reduced-resolution image/transparency mask (5 = 0x5) │
│        11 │ AdobeDeflate       │             │ transparency mask          │ single image plane   │ unsigned integer │                   │ none 1 (0x1) │ reduced-resolution image/transparency mask (5 = 0x5) │
│        12 │ AdobeDeflate       │             │ transparency mask          │ single image plane   │ unsigned integer │                   │ none 1 (0x1) │ reduced-resolution image/transparency mask (5 = 0x5) │
│        13 │ AdobeDeflate       │             │ transparency mask          │ single image plane   │ unsigned integer │                   │ none 1 (0x1) │ reduced-resolution image/transparency mask (5 = 0x5) │
│        14 │ AdobeDeflate       │             │ transparency mask          │ single image plane   │ unsigned integer │                   │ none 1 (0x1) │ reduced-resolution image/transparency mask (5 = 0x5) │
├───────────┴────────────────────┴─────────────┴────────────────────────────┴──────────────────────┴──────────────────┴───────────────────┴──────────────┴──────────────────────────────────────────────────────┤
│ 14 rows                                                                                                                                                                                             9 columns │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.