digitalearthafrica / config Goto Github PK

View Code? Open in Web Editor NEW

8.0 8.0 11.0 1.35 MB

Config files for datacube and associated applications

License: Apache License 2.0

Python 39.96% Dockerfile 0.05% Shell 6.67% Scheme 53.32%

hacktoberfest

config's People

Contributors

Stargazers

Watchers

Forkers

gypsybojangles spacemanpaul eloise-b andrewdhicks zacharlie flexigrobots-h2020 nf-s steve9164 indiphile vikineema

config's Issues

Deploy dataset_cache_rules across all reslim

Add dataset_cache_rules to each reslim config.

Implements progressive cache rules. They cause OWS to output cache-control headers on GetMap requests. These headers tell Cloudfront how long to cache each response. The idea is that more expensive queries (i.e. queries that hit many datasets) get cached for longer, while requests that are fast and cheap anyway only get cached for a shorter amount of time.
The overall effect on user experience is quite noticeable - especially on products that do not have a low-resolution summary mosaic.

More details here: https://datacube-ows.readthedocs.io/en/latest/cfg_layers.html#dataset-cache-rules

Combine product and ows testing

Currently there are two CI jobs that can be combined.

The product-lists-test one adds all our products to a DB.

The ows-config-test-build does linting, testing and builds the config Docker image.

We should combine both of those into one test, which includes the following tasks:

Create a single Docker Compose file with postgres, ows and index containers
For each of DEV and PROD, the CI should add all the products, run the config test and then tear down the environment.

Should make things better!

adding s1_rtc to ows

Adding Sentinel to ows.

Define S1 display styles.

Need to add CRS elements for WCS and WMS

From @SpacemanPaul

Latest merge to OWS master moves the "native_crs" and "native_resolution" elements out of WCS and into the main layer config. (As they will be needed for WMS as well under the new OWS resource management framework.) This may produce warning messages in dev.

Note that both elements are only required if the native CRS and resolution cannot be determined from the product metadata. If you do have layers that need them in config, you can specify them in BOTH positions (i.e. in wcs section AND main layer section) and then it will work with both current master and 1.8.15. (It will throw a warning in master is all). (edited)

Adding JERS mosaic

The JERS mosaic (currently product "jers_mosaic", but maybe reindexed to be "jers_sar_mosaic") should be added to ows_cfg.py using information from the metadata catalog. Only one display style for hh is needed. Jers hh digital numbers have a scaling ~1.21 (10^0.083) times ALOS hh digital numbers. Therefore the alos_hh color ramp with all values x 1.21 will work.

ows - wofs summary legend

This legend label for annual summary has been broken for a while:
https://ows.digitalearth.africa/legend/wofs_2_annual_summary_frequency/WOfS_frequency/legend.png

More importantly, we should fix it before new wofs annual summary becomes available:
https://ows.dev.digitalearth.africa/legend/wofs_ls_summary_annual/wofs_summary_annual_frequency/legend.png

Landsat 8 geomedian styling

The provisional landsat 8 (collection 2) data has a scaling and an offset, so Surface Reflectance = DN * 2.75e-5 - 0.2. The default band indices calculations are therefore incorrect.
The NDVI/NDWI/MNDWI layers for the Landsat 8 Geomedian should be removed or updated (using customized functions).

`rainfall_chirps_daily` to OWS

rainfall_chirps_daily is indexed to the DE Africa ODC and needs OWS configuration to be available in DE Africa Maps and through GIS software.

As a new style will need to be defined it might be good to split the current single-file config into the 3-file band/style/layer structure. This will also make it easier to include CHPclim in the future...

CRS changed from EPSG:102022 to ESRI:102022

Change to product definition crs for ls_usgs_wofs_summary to ESRI:102022

[2020-04-02 23:10:13,690] datacube_ows.ows_configuration [6ae209ed-29ce-4a25-af24-05571fbd5d56] [ERROR] Could not load layer: Native CRS for product ls_usgs_wofs_summary (ESRI:102022) not in published CRSs
[2020-04-02 23:10:13,699] datacube_ows.ows_configuration [6ae209ed-29ce-4a25-af24-05571fbd5d56] [ERROR] Could not load layer: Native CRS for product ls_usgs_wofs_summary (ESRI:102022) not in published CRSs
[2020-04-02 23:10:13,708] datacube_ows.ows_configuration [6ae209ed-29ce-4a25-af24-05571fbd5d56] [ERROR] Could not load layer: Native CRS for product ls_usgs_wofs_summary (ESRI:102022) not in published CRSs
[2020-04-02 23:10:13,714] datacube_ows.ows_configuration [2a0d5da7-aef7-41cb-82a9-2769df11123d] [ERROR] Could not load layer: Native CRS for product ls_usgs_wofs_summary (ESRI:102022) not in published CRSs
[2020-04-02 23:10:13,716] datacube_ows.ows_configuration [6ae209ed-29ce-4a25-af24-05571fbd5d56] [ERROR] Could not load layer: Native CRS for product ls_usgs_wofs_summary (ESRI:102022) not in published CRSs
[2020-04-02 23:10:13,724] datacube_ows.ows_configuration [2a0d5da7-aef7-41cb-82a9-2769df11123d] [ERROR] Could not load layer: Native CRS for product ls_usgs_wofs_summary (ESRI:102022) not in published CRSs
[2020-04-02 23:10:13,732] datacube_ows.ows_configuration [2a0d5da7-aef7-41cb-82a9-2769df11123d] [ERROR] Could not load layer: Native CRS for product ls_usgs_wofs_summary (ESRI:102022) not in published CRSs
[2020-04-02 23:10:13,740] datacube_ows.ows_configuration [2a0d5da7-aef7-41cb-82a9-2769df11123d] [ERROR] Could not load layer: Native CRS for product ls_usgs_wofs_summary (ESRI:102022) not in published CRSs

Sentinel-2 SCL through web service

Currently this classification is not visible on the map portal. Despite its limitations, this layer contains some useful information derived from Sentinel-2 data. This is not high priority but good to have.

ows refactored inconsistencies compare to ows africa

issue	ows refactor	ows africa	action - accept difference or fix
1. band values	link	link	- [x] Accept - [x] Fix
2. band values	link	link	- [x] Fixed
3. band values	link	link	- [x] Fixed
4. band values	link	link	- [x] Fixed
5. derived band values	link	link	- [x] Fixed
6. image size	link	link	- [x] Fixed
7. image size	link	link	- [x] Fixed
8. image size	link	link	- [x] Accept - [x] Fix
9. image size	link	link	- [x] Fixed test case
10. image size	link	link	- [x] Fixed
11. image size	link	link	- [x] Fixed
12. Derived band	link	link	- [x] Fixed

The submit issue button in the DE Africa sandbox link to the DE Australia Github

misleading docker build folder name

https://github.com/digitalearthafrica/config/blob/master/Dockerfile#L3

this should be deafrica-config

prototype crop mask ows

Add prototype crop_mask_eastern product to dev ows.

Product in africa dev: crop_mask_eastern
bands: mask, prob

Display style:
mask band: green/yellow for 1, transparent for 0
(e.g. similar to https://croplands.org/app/map?lat=1.62576&lng=36.07910156250001&zoom=6)
reversed mask: grey for 0, transparent for 1
prob band: plasma or magma for a range 50 -100.

Add CGLS land cover to OWS

Add CGLS land cover to OWS, using styles defined in https://github.com/digitalearthafrica/deafrica-sandbox-notebooks/blob/7823bc167db38d93d76f495cdd7518314677332f/Tools/deafrica_tools/plotting.py#L1114

Avoid building docker container on product definition changes

Docker config containers are built on product yaml changes. Avoid this.

ALOS styling

To improve ALOS styling in ows_cfg.py

For hh_hv_hv_over_hh, apply following stretching (scale range?) per-band:

hh: [500, 10000]
hv: [200, 4000]
hv/hh:  [0.1, 1.0]

style_alos_hh_over_hv is not necessary.
If possible, add a Radar Vegetation Index band, which is (4*hv^2)/(hh^2+hv^2), with scale range [0.2, 1] and 'viridis' colormap.

Reduce max datasets for WCS

Most product limits set at 64. This is causing load issues when using WCS in GIS software, especially for users using less complex queries.

Limit of around 24-32 would be reasonable for most products.

Config set in https://github.com/digitalearthafrica/config/blob/master/services/ows_refactored/common/ows_reslim_cfg.py

Update coastlines layer setting

update abstract and contact for coastlines

Broken link in FC legend

config/services/ows.py

Line 2928 in 3b04fce

 "url": "https://data.dea.ga.gov.au/fractional-cover/fc-percentile/annual/v2.1.0/fcp_legend.png", 

Additional band indices for S2

A few users have expressed interest to use Map to look at more indices for S2 data. The product dev team will look into a few of them and provide appropriate styles.
A short list to start with:

water quality (Chlorophyll) - done.
TSS water quality
urban
NBR

Add DEM derivative layers to prod ows.

This product is already in DEV. Add it to prod ows.

Semiannual GeoMAD low-res tiles

Low res tiling allows preview at continental scale on Terria Map
Exists for the Annual GeoMAD, was run by Kirill
Significantly improves user experience and allows for full range of data preview at any zoom level (see Annual GeoMAD for example)

Loading "dem_srtm" fails for "EPSG:32639" query

The DE Africa Coastlines process loads data from the "dem_srtm" product to mask out areas of possible terrain shadow. However, when I attempt to load data from "dem_srtm" over an area of eastern Somalia using coordinates in the "EPSG:32639" CRS, this fails:

lon1, lat1 = 382157.79098872, 929883.30909207
lon2, lat2 = 417776.30111789, 901707.94360185

dc.load(product="dem_srtm", 
        x=(lon1, lon2),
        y=(lat1, lat2),
        crs="EPSG:32639")

Error:

IllegalArgumentException: CGAlgorithmsDD::orientationIndex encountered NaN/Inf numbers
---------------------------------------------------------------------------
PredicateError                            Traceback (most recent call last)
/usr/local/lib/python3.8/dist-packages/shapely/predicates.py in __call__(self, this, other, *args)
     14         try:
---> 15             return self.fn(this._geom, other._geom, *args)
     16         except PredicateError as err:

/usr/local/lib/python3.8/dist-packages/shapely/geos.py in errcheck_predicate(result, func, argtuple)
    601     if result == 2:
--> 602         raise PredicateError("Failed to evaluate %s" % repr(func))
    603     return result

PredicateError: Failed to evaluate <_FuncPtr object at 0x7f6316c94f40>

During handling of the above exception, another exception occurred:

TopologicalError                          Traceback (most recent call last)
/tmp/ipykernel_292/1842874794.py in <module>
      2 lon2, lat2 = 417776.30111789, 901707.94360185
      3 
----> 4 dc.load(product="dem_srtm", 
      5         x=(lon1, lon2),
      6         y=(lat1, lat2),

/usr/local/lib/python3.8/dist-packages/datacube/api/core.py in load(self, product, measurements, output_crs, resolution, resampling, skip_broken_datasets, dask_chunks, like, fuse_func, align, datasets, dataset_predicate, progress_cbk, **query)
    384 
    385         if datasets is None:
--> 386             datasets = self.find_datasets(product=product,
    387                                           like=like,
    388                                           ensure_location=True,

/usr/local/lib/python3.8/dist-packages/datacube/api/core.py in find_datasets(self, **search_terms)
    446         .. seealso:: :meth:`group_datasets` :meth:`load_data` :meth:`find_datasets_lazy`
    447         """
--> 448         return list(self.find_datasets_lazy(**search_terms))
    449 
    450     def find_datasets_lazy(self, limit=None, ensure_location=False, dataset_predicate=None, **kwargs):

/usr/local/lib/python3.8/dist-packages/datacube/api/core.py in <genexpr>(.0)
    472 
    473         if ensure_location:
--> 474             datasets = (dataset for dataset in datasets if dataset.uris)
    475 
    476         # If a predicate function is provided, use this to filter datasets before load

/usr/local/lib/python3.8/dist-packages/datacube/api/core.py in select_datasets_inside_polygon(datasets, polygon)
    879     query_crs = polygon.crs
    880     for dataset in datasets:
--> 881         if intersects(polygon, dataset.extent.to_crs(query_crs)):
    882             yield dataset
    883 

/usr/local/lib/python3.8/dist-packages/datacube/utils/geometry/_base.py in intersects(a, b)
   1277     """ Returns True if geometries intersect, else False
   1278     """
-> 1279     return a.intersects(b) and not a.touches(b)
   1280 
   1281 

/usr/local/lib/python3.8/dist-packages/datacube/utils/geometry/_base.py in wrapped(*args)
    365                 raise CRSMismatchError((first.crs, arg.crs))
    366 
--> 367         result = method(*[arg.geom for arg in args])
    368         if isinstance(result, base.BaseGeometry):
    369             return Geometry(result, first.crs)

/usr/local/lib/python3.8/dist-packages/datacube/utils/geometry/_base.py in intersects(self, other)
    467     @wrap_shapely
    468     def intersects(self, other: 'Geometry') -> bool:
--> 469         return self.intersects(other)
    470 
    471     @wrap_shapely

/usr/local/lib/python3.8/dist-packages/shapely/geometry/base.py in intersects(self, other)
    797     def intersects(self, other):
    798         """Returns True if geometries intersect, else False"""
--> 799         return bool(self.impl['intersects'](self, other))
    800 
    801     def overlaps(self, other):

/usr/local/lib/python3.8/dist-packages/shapely/predicates.py in __call__(self, this, other, *args)
     16         except PredicateError as err:
     17             # Dig deeper into causes of errors.
---> 18             self._check_topology(err, this, other)
     19 
     20 

/usr/local/lib/python3.8/dist-packages/shapely/topology.py in _check_topology(self, err, *geoms)
     35         for geom in geoms:
     36             if not geom.is_valid:
---> 37                 raise TopologicalError(
     38                     "The operation '%s' could not be performed. "
     39                     "Likely cause is invalidity of the geometry %s" % (

TopologicalError: The operation 'GEOSIntersects_r' could not be performed. Likely cause is invalidity of the geometry <shapely.geometry.polygon.Polygon object at 0x7f630d037f40>

This seems to line up with an area that failed in previous DEA Coastlines runs (e.g. blue tiles here), which itself seems to line up with UTM Zone 39.

Environment:
DE Africa Dev Sandbox, datacube==0.8.6

Bulk Styling handler in OWS Styling API

Aiming for optimizing the styling tool to do bulk styling Xarray with multiple timestamps.