Moving from CDO to xarray+dask about ecmean4 HOT 7 CLOSED

Another issue which seems to be there is that xESMF, actually our preferred option, does not have support for unstructured grids so that it can't be used for gaussian reduced grids pangeo-data/xESMF#115

from ecmean4.

New commit 4dd03ca introduces also xr_performance_indices.py and solves a few issues identified in the past.

Positive aspects:

• Also performance indices code is 3x faster, even without dask.
• Horizontal Interpolation is carried out with xESMF, which supports the NEMO curvilinear grid.
• Vertical interpolation can be done natively with Xarray.
• Metpy handles units also for axis simplifying pressure level conversion

Negative aspects:

• Interpolating the gaussian reduced is a bit of pain, and a two step interpolation expanding the reduced to the regular needs to be carried out.
• Area and vertical averages needs precomputed weights, which has to be done manually.
• Xarray is extremely picky in file formats. This has to be formatted in a common way otherwise it gets mad at mixing grids. The current solution is filled with ad-hoc correction which are not suited for a code generalization.

Overall, a lot of work needs to be done to make it more general. The dictionary used to transfer the mask and remap weights need to be restructured. Also climatology files needs to be fixed in a better way.

from ecmean4.

Commit 4b2ec64:

• automatic adjustment of climatological files
• support for EC22 climatology (missing siconc)

Once fixed the siconc issues (with NaN resulting from variance 0 in EC22) we need to test it against CMIP files. It might be required to write a "cmorizator" of the lon/lat/time/lev names so that we have a common standard for dimensions. so fare it is quite weird, having lon,lat,time_counter and pressure_levels (derived from the names of EC-Earth4).

from ecmean4.

Commit 1924f49:

• fixing issue with 0 variance in EC22
• refactoring of the supporting functions into a single xr_ecmean.py file, also duplicating useful functions from older CDO based ecmean.py.

from ecmean4.

Commit e9a126b and previous:

• Support for CMIP6 data in xr_global_mean.py, tested on a set of models.
• Fixing a bug in CMIP6 interface file (see #41)
• Extending the options for the area_cell function (but still a many possibilities not covered)

It turns out that dask shows some conflict with the multiprocessing package which needs to be investigated. This can lead to weird freeze of the running code, or even abrupt crash. A decent solution is to set no parallelization in open_mfdataset call via

import dask
dask.config.set(scheduler="synchronous")

from ecmean4.

Commit 862ddff and previous:

• Support for xr_performance_indices.py, teste on a few models
• Introducing a xr_preproc() function to harmonized metadata in cmor-like (lon,lat,plev) structure.

The new version thus ready to create a draft pull request: I will document missing features, especially regarding a necessary harmonization of area weights and interpolation

from ecmean4.

This can be closed following the merge of the pull request

from ecmean4.