digitalearthafrica / config Goto Github PK
View Code? Open in Web Editor NEWConfig files for datacube and associated applications
License: Apache License 2.0
Config files for datacube and associated applications
License: Apache License 2.0
Add dataset_cache_rules
to each reslim
config.
Implements progressive cache rules. They cause OWS to output cache-control headers on GetMap requests. These headers tell Cloudfront how long to cache each response. The idea is that more expensive queries (i.e. queries that hit many datasets) get cached for longer, while requests that are fast and cheap anyway only get cached for a shorter amount of time.
The overall effect on user experience is quite noticeable - especially on products that do not have a low-resolution summary mosaic.
More details here: https://datacube-ows.readthedocs.io/en/latest/cfg_layers.html#dataset-cache-rules
Currently there are two CI jobs that can be combined.
The product-lists-test
one adds all our products to a DB.
The ows-config-test-build
does linting, testing and builds the config Docker image.
We should combine both of those into one test, which includes the following tasks:
Should make things better!
Adding Sentinel to ows.
Define S1 display styles.
From @SpacemanPaul
Latest merge to OWS master moves the "native_crs" and "native_resolution" elements out of WCS and into the main layer config. (As they will be needed for WMS as well under the new OWS resource management framework.) This may produce warning messages in dev.
Note that both elements are only required if the native CRS and resolution cannot be determined from the product metadata. If you do have layers that need them in config, you can specify them in BOTH positions (i.e. in wcs section AND main layer section) and then it will work with both current master and 1.8.15. (It will throw a warning in master is all). (edited)
The JERS mosaic (currently product "jers_mosaic", but maybe reindexed to be "jers_sar_mosaic") should be added to ows_cfg.py using information from the metadata catalog. Only one display style for hh is needed. Jers hh digital numbers have a scaling ~1.21 (10^0.083) times ALOS hh digital numbers. Therefore the alos_hh color ramp with all values x 1.21 will work.
This legend label for annual summary has been broken for a while:
https://ows.digitalearth.africa/legend/wofs_2_annual_summary_frequency/WOfS_frequency/legend.png
More importantly, we should fix it before new wofs annual summary becomes available:
https://ows.dev.digitalearth.africa/legend/wofs_ls_summary_annual/wofs_summary_annual_frequency/legend.png
The provisional landsat 8 (collection 2) data has a scaling and an offset, so Surface Reflectance = DN * 2.75e-5 - 0.2. The default band indices calculations are therefore incorrect.
The NDVI/NDWI/MNDWI layers for the Landsat 8 Geomedian should be removed or updated (using customized functions).
rainfall_chirps_daily
is indexed to the DE Africa ODC and needs OWS configuration to be available in DE Africa Maps and through GIS software.
As a new style will need to be defined it might be good to split the current single-file config into the 3-file band/style/layer structure. This will also make it easier to include CHPclim in the future...
Change to product definition crs for ls_usgs_wofs_summary to ESRI:102022
[2020-04-02 23:10:13,690] datacube_ows.ows_configuration [6ae209ed-29ce-4a25-af24-05571fbd5d56] [ERROR] Could not load layer: Native CRS for product ls_usgs_wofs_summary (ESRI:102022) not in published CRSs
[2020-04-02 23:10:13,699] datacube_ows.ows_configuration [6ae209ed-29ce-4a25-af24-05571fbd5d56] [ERROR] Could not load layer: Native CRS for product ls_usgs_wofs_summary (ESRI:102022) not in published CRSs
[2020-04-02 23:10:13,708] datacube_ows.ows_configuration [6ae209ed-29ce-4a25-af24-05571fbd5d56] [ERROR] Could not load layer: Native CRS for product ls_usgs_wofs_summary (ESRI:102022) not in published CRSs
[2020-04-02 23:10:13,714] datacube_ows.ows_configuration [2a0d5da7-aef7-41cb-82a9-2769df11123d] [ERROR] Could not load layer: Native CRS for product ls_usgs_wofs_summary (ESRI:102022) not in published CRSs
[2020-04-02 23:10:13,716] datacube_ows.ows_configuration [6ae209ed-29ce-4a25-af24-05571fbd5d56] [ERROR] Could not load layer: Native CRS for product ls_usgs_wofs_summary (ESRI:102022) not in published CRSs
[2020-04-02 23:10:13,724] datacube_ows.ows_configuration [2a0d5da7-aef7-41cb-82a9-2769df11123d] [ERROR] Could not load layer: Native CRS for product ls_usgs_wofs_summary (ESRI:102022) not in published CRSs
[2020-04-02 23:10:13,732] datacube_ows.ows_configuration [2a0d5da7-aef7-41cb-82a9-2769df11123d] [ERROR] Could not load layer: Native CRS for product ls_usgs_wofs_summary (ESRI:102022) not in published CRSs
[2020-04-02 23:10:13,740] datacube_ows.ows_configuration [2a0d5da7-aef7-41cb-82a9-2769df11123d] [ERROR] Could not load layer: Native CRS for product ls_usgs_wofs_summary (ESRI:102022) not in published CRSs
Currently this classification is not visible on the map portal. Despite its limitations, this layer contains some useful information derived from Sentinel-2 data. This is not high priority but good to have.
issue | ows refactor | ows africa | action - accept difference or fix |
---|---|---|---|
1. band values | link | link | - [x] Accept - [x] Fix |
2. band values | link | link | - [x] Fixed |
3. band values | link | link | - [x] Fixed |
4. band values | link | link | - [x] Fixed |
5. derived band values | link | link | - [x] Fixed |
6. image size | link | link | - [x] Fixed |
7. image size | link | link | - [x] Fixed |
8. image size | link | link | - [x] Accept - [x] Fix |
9. image size | link | link | - [x] Fixed test case |
10. image size | link | link | - [x] Fixed |
11. image size | link | link | - [x] Fixed |
12. Derived band | link | link | - [x] Fixed |
https://github.com/digitalearthafrica/config/blob/master/Dockerfile#L3
this should be deafrica-config
Add prototype crop_mask_eastern product to dev ows.
Product in africa dev: crop_mask_eastern
bands: mask, prob
Display style:
mask band: green/yellow for 1, transparent for 0
(e.g. similar to https://croplands.org/app/map?lat=1.62576&lng=36.07910156250001&zoom=6)
reversed mask: grey for 0, transparent for 1
prob band: plasma or magma for a range 50 -100.
Add CGLS land cover to OWS, using styles defined in https://github.com/digitalearthafrica/deafrica-sandbox-notebooks/blob/7823bc167db38d93d76f495cdd7518314677332f/Tools/deafrica_tools/plotting.py#L1114
Docker config containers are built on product yaml changes. Avoid this.
To improve ALOS styling in ows_cfg.py
hh: [500, 10000]
hv: [200, 4000]
hv/hh: [0.1, 1.0]
Most product limits set at 64
. This is causing load issues when using WCS in GIS software, especially for users using less complex queries.
Limit of around 24-32 would be reasonable for most products.
Config set in https://github.com/digitalearthafrica/config/blob/master/services/ows_refactored/common/ows_reslim_cfg.py
update abstract and contact for coastlines
Line 2928 in 3b04fce
A few users have expressed interest to use Map to look at more indices for S2 data. The product dev team will look into a few of them and provide appropriate styles.
A short list to start with:
This product is already in DEV. Add it to prod ows.
The DE Africa Coastlines process loads data from the "dem_srtm" product to mask out areas of possible terrain shadow. However, when I attempt to load data from "dem_srtm" over an area of eastern Somalia using coordinates in the "EPSG:32639" CRS, this fails:
lon1, lat1 = 382157.79098872, 929883.30909207
lon2, lat2 = 417776.30111789, 901707.94360185
dc.load(product="dem_srtm",
x=(lon1, lon2),
y=(lat1, lat2),
crs="EPSG:32639")
Error:
IllegalArgumentException: CGAlgorithmsDD::orientationIndex encountered NaN/Inf numbers
---------------------------------------------------------------------------
PredicateError Traceback (most recent call last)
/usr/local/lib/python3.8/dist-packages/shapely/predicates.py in __call__(self, this, other, *args)
14 try:
---> 15 return self.fn(this._geom, other._geom, *args)
16 except PredicateError as err:
/usr/local/lib/python3.8/dist-packages/shapely/geos.py in errcheck_predicate(result, func, argtuple)
601 if result == 2:
--> 602 raise PredicateError("Failed to evaluate %s" % repr(func))
603 return result
PredicateError: Failed to evaluate <_FuncPtr object at 0x7f6316c94f40>
During handling of the above exception, another exception occurred:
TopologicalError Traceback (most recent call last)
/tmp/ipykernel_292/1842874794.py in <module>
2 lon2, lat2 = 417776.30111789, 901707.94360185
3
----> 4 dc.load(product="dem_srtm",
5 x=(lon1, lon2),
6 y=(lat1, lat2),
/usr/local/lib/python3.8/dist-packages/datacube/api/core.py in load(self, product, measurements, output_crs, resolution, resampling, skip_broken_datasets, dask_chunks, like, fuse_func, align, datasets, dataset_predicate, progress_cbk, **query)
384
385 if datasets is None:
--> 386 datasets = self.find_datasets(product=product,
387 like=like,
388 ensure_location=True,
/usr/local/lib/python3.8/dist-packages/datacube/api/core.py in find_datasets(self, **search_terms)
446 .. seealso:: :meth:`group_datasets` :meth:`load_data` :meth:`find_datasets_lazy`
447 """
--> 448 return list(self.find_datasets_lazy(**search_terms))
449
450 def find_datasets_lazy(self, limit=None, ensure_location=False, dataset_predicate=None, **kwargs):
/usr/local/lib/python3.8/dist-packages/datacube/api/core.py in <genexpr>(.0)
472
473 if ensure_location:
--> 474 datasets = (dataset for dataset in datasets if dataset.uris)
475
476 # If a predicate function is provided, use this to filter datasets before load
/usr/local/lib/python3.8/dist-packages/datacube/api/core.py in select_datasets_inside_polygon(datasets, polygon)
879 query_crs = polygon.crs
880 for dataset in datasets:
--> 881 if intersects(polygon, dataset.extent.to_crs(query_crs)):
882 yield dataset
883
/usr/local/lib/python3.8/dist-packages/datacube/utils/geometry/_base.py in intersects(a, b)
1277 """ Returns True if geometries intersect, else False
1278 """
-> 1279 return a.intersects(b) and not a.touches(b)
1280
1281
/usr/local/lib/python3.8/dist-packages/datacube/utils/geometry/_base.py in wrapped(*args)
365 raise CRSMismatchError((first.crs, arg.crs))
366
--> 367 result = method(*[arg.geom for arg in args])
368 if isinstance(result, base.BaseGeometry):
369 return Geometry(result, first.crs)
/usr/local/lib/python3.8/dist-packages/datacube/utils/geometry/_base.py in intersects(self, other)
467 @wrap_shapely
468 def intersects(self, other: 'Geometry') -> bool:
--> 469 return self.intersects(other)
470
471 @wrap_shapely
/usr/local/lib/python3.8/dist-packages/shapely/geometry/base.py in intersects(self, other)
797 def intersects(self, other):
798 """Returns True if geometries intersect, else False"""
--> 799 return bool(self.impl['intersects'](self, other))
800
801 def overlaps(self, other):
/usr/local/lib/python3.8/dist-packages/shapely/predicates.py in __call__(self, this, other, *args)
16 except PredicateError as err:
17 # Dig deeper into causes of errors.
---> 18 self._check_topology(err, this, other)
19
20
/usr/local/lib/python3.8/dist-packages/shapely/topology.py in _check_topology(self, err, *geoms)
35 for geom in geoms:
36 if not geom.is_valid:
---> 37 raise TopologicalError(
38 "The operation '%s' could not be performed. "
39 "Likely cause is invalidity of the geometry %s" % (
TopologicalError: The operation 'GEOSIntersects_r' could not be performed. Likely cause is invalidity of the geometry <shapely.geometry.polygon.Polygon object at 0x7f630d037f40>
This seems to line up with an area that failed in previous DEA Coastlines runs (e.g. blue tiles here), which itself seems to line up with UTM Zone 39.
Environment:
DE Africa Dev Sandbox, datacube==0.8.6
Aiming for optimizing the styling tool to do bulk styling Xarray with multiple timestamps.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.