comet-toolkit / obsarray Goto Github PK

Hi @shunt16 and @pdevis,

I am a researcher working with EO imagery and uncertainty analyses, and I am wondering if your current development plan includes the support for spatio-temporal auto-correlations in different coordinate reference systems? This is a feature that really seems key to EO these days (many researchers analyze the spatial/temporal sum/mean of a single type of observation in EO data without accounting for the fact that uncertainties are far from being independent).

If it is something that you think would fit well in obsarray, and you're looking for some help, I'd be happy to try to contribute! 😄
I am looking to code this consistently somewhere to apply to any type of EO data.

Some background: I use a lot of satellite imagery time series, that often comes with noise/alignment issues. The effect on uncertainty can be nicely estimated by spatial/temporal autocorrelation functions (I've been relying on scikit-gstat for variography).

I've worked a lot with xarray, but also rasterio/geopandas for raster/vector data. We've been trying to develop open-source efforts to simplify geospatial analysis: https://github.com/GlacioHack/geoutils and https://github.com/GlacioHack/xdem, still very much in development (we intend to shift towards xarray through rioxarray in the next version, as rioxarray wasn't stable yet when we started putting the package together).

In any case, thanks a lot for this nice effort! 🙂

https://docs.python.org/3/whatsnew/3.12.html#configparser
warner/python-spake2#13
pydata/pandas-datareader#969

I think getting access to what flags are currently set is very useful. This is currently possible with:

from obsarray.templater.dataset_util import DatasetUtil
print([DatasetUtil.get_set_flags(flag) for flag in data["quality_flag"]])

However I think this functionality is so useful that it would be great if it would be more easily accessible. Maybe we can write a little wrapper around it?

implement the correlation forms from https://research.reading.ac.uk/fiduceo/cdr/theoretical-basis-3/4-completing-the-effects-table/

Especially the exponential decay and triangular relative would be useful as they are needed in MACRAD project.

Some obsarray functionality raises an error for uncertainty variables where the unc_comp attribute is a string rather than a list. This can arise when there is only one uncertainty component for a variable. Obsarray should be able to handle unc_comps as a list or a string.

e.g.
line run:

del ds.unc[obs_var][var]

error:
Traceback (most recent call last):
File "C:\Users\mes\Anaconda3\envs\eoio\lib\site-packages\IPython\core\interactiveshell.py", line 3508, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 1, in
del ds.unc[obs_var][var]
File "c:\users\mes\code\pycharmprojects\obsarray\obsarray\unc_accessor.py", line 424, in delitem
self._obj.unc._remove_unc_var(self._var_name, unc_var)
File "c:\users\mes\code\pycharmprojects\obsarray\obsarray\unc_accessor.py", line 898, in _remove_unc_var
self._obj[obs_var].attrs["unc_comps"].remove(unc_var)
AttributeError: 'str' object has no attribute 'remove'

sphinx-autosummary-accessors is a sphinx extension to properly document pandas or xarray style accessors using autosummary and autodoc. Should improve the docs to make use of this.

In my use case in hypernets, I don't need to just rename an uncertainty variable, but also the variable itself, as well as error correlation matrices.
Typically, the name of the variable (e.g.) temperature is also present in the uncertainty variable (e.g. u_ran_temperature) and name of err_corr_matrix (err_corr_temperature). I would like a method (see my previous suggestion for example code) in which we can replace all instances of a certain string (in this case "temperature") by a different string (e.g. "temperature_SWIR") in either variable name, unc_variable name or erro corr matrix variable name.

A function to rename a variable, or a dimension would be useful, which then also renames all the uncertainty variables, and associated attributes etc in the relevant uncertainty and err_corr variables. Something along the line of:

def rename_var(self, dataset, varname, varname_new, wavvar, wavvar_new):
    replace_dict = {wavvar: wavvar_new}
    for var in dataset.variables:
        if varname in var:
            replace_dict[var] = var.replace(varname, varname_new)
    dataset_temp = dataset.rename(replace_dict)
    if "unc_comps" in dataset_temp[varname_new].attrs.keys():
        dataset_temp[varname_new].attrs["unc_comps"] = [
            comp.replace(varname, varname_new)
            for comp in dataset_temp[varname_new].attrs["unc_comps"]
        ]
        for comp in dataset_temp[varname_new].attrs["unc_comps"]:
            for i in range(2):
                if wavvar == dataset_temp[comp].attrs["err_corr_%s_dim"%(i+1)]:
                    dataset_temp[comp].attrs["err_corr_%s_dim"%(i+1)]=wavvar_new
                if len(dataset_temp[comp].attrs["err_corr_%s_params"%(i+1)]) > 0:
                    if "err_corr" in dataset_temp[comp].attrs["err_corr_%s_params"%(i+1)][0]:
                        dataset_temp[comp].attrs["err_corr_%s_params"%(i+1)][0] = \
                        dataset_temp[comp].attrs["err_corr_%s_params"%(i+1)][0].replace(varname, varname_new)
    return dataset_temp 

Reading in Sentinel-3 quality flags netcdf with rioxarray, stores the flag_masks attribute as a np.ndarray which throws up an error when trying to set the flag masks in unpack_flag_attrs in obsarray\templater\dataset_util.py:

flag_mask = attrs["flag_masks"].split(",") if "flag_masks" in attrs else []
flag_mask = [int(m) for m in flag_mask] if flag_mask != [""] else []

AttributeError: 'numpy.ndarray' object has no attribute 'split'

similar to renaming unc variable, but for err_corr matrices (when a named variable)

Some easy handling of scallar variables (with uncertainties but no error_correlation) would be useful

Hello,

I am starting to use punpy and obsarray as part of an ESA project. I am now at the stage where I want to create my first digital effects table. I ran the example linked from the CoMet pages, on google collab:

https://colab.research.google.com/github/comet-toolkit/comet_training/blob/main/defining_digital_effects_table.ipynb

The notebook runs without errors, but I do not understand the resulting obsarray dataset. I do not see that it contains any obsarray-specific attributes to link uncertainty variables together and describe the uncertainty model contained in the effects table.

This is the ncdump -ch that I get when running (a local copy of) the notebook and save the effects table in the last cell (the "more realistic example").

This is with obsarray v0.4.2 and xarray 2023.8.0:

# store example file
ds.to_netcdf("/home/thomasl/Downloads/obsarray_test.nc")
!ncdump -ch /home/thomasl/Downloads/obsarray_test.nc

shows:

netcdf obsarray_test {
dimensions:
	x = 20 ;
	y = 30 ;
	time = 6 ;
variables:
	double temperature(x, y, time) ;
		temperature:_FillValue = 9.96920996838687e+36 ;
		temperature:units = "K" ;
		temperature:u_components = "u_temperature" ;
	double u_ran_temperature(x, y, time) ;
		u_ran_temperature:_FillValue = 9.96920996838687e+36 ;
		u_ran_temperature:units = "K" ;
	double u_sys_temperature(x, y, time) ;
		u_sys_temperature:_FillValue = 9.96920996838687e+36 ;
	double pressure(x, y, time) ;
		pressure:_FillValue = 9.96920996838687e+36 ;
	double u_str_pressure(x, y, time) ;
		u_str_pressure:_FillValue = 9.96920996838687e+36 ;
	double err_corr_str_pressure_y(y, y) ;
		err_corr_str_pressure_y:_FillValue = 9.96920996838687e+36 ;
	double n_moles(x, y, time) ;
		n_moles:_FillValue = 9.96920996838687e+36 ;
	double u_ran_n_moles(x, y, time) ;
		u_ran_n_moles:_FillValue = 9.96920996838687e+36 ;

// global attributes:
		:obsarray_version = "v0.2.4" ;
		:xarray_version = "2023.8.0" ;
}

Is this as expected? As you see there are no obsarray specific attributes such as err_corr_1_form = "random", etc...