Hi there! I recently heard about this project via a colleague watching your presentation at ESIP.

I am the lead developer of fsspec and the fsspec-reference-maker. You might find the following articles interesting:

• https://medium.com/pangeo/fake-it-until-you-make-it-reading-goes-netcdf4-data-on-aws-s3-as-zarr-for-rapid-data-access-61e33f8fe685
• https://medium.com/pangeo/cloud-performant-netcdf4-hdf5-with-zarr-fsspec-and-intake-3d3a3e7cb935

After only a brief perusal of this repo, the following comparisons come to mind.

Things we can learn from you (there are probably more!)

• combining adjacent reads (like corot project); this is tractable, but less important given that our reads are concurrent
• caching redirects from auth mechanism

Things we do that you might find interesting in our project

• combining many datasets into singe ensemble datasets
• not just HDF5, but also grib2 and Tiff and more to come
• any storage backend supported by fsspec (s3, gcs, azure, http, ftp, ..., see here); references and data can live separately. Could be made to work with data living on different storage backends but part of a single aggregate data set.
• concurrent access to chunks for S3, Http, GCS, abfs
• explicit ability to be serialised for distributed processing with dask
• requires no extra installs aside from standard xarray, zarr, fsspec
• doesn't necessarily need xarray; is intended to be multi-language
• json, parquet or zarr storage of the reference metadata files

Let's work together and not invent more wheels!

I have a NetCDF file that produced a dmrpp file with the following compressionType:

<dmrpp:chunks compressionType="deflate shuffle" byteOrder="LE">

This results in EosdisStore throwing an Unrecognized CompressionType error. Full traceback in "Details" below, but it's pretty clear and self-explanatory. Looks like you already handle the deflate case, but dmrpp.py needs to be revised to also handle the deflate + shuffle case.

The source file is the first tavg1_2d_flx file from here (direct file link), with the .dmrpp created via the mkdmrpp script in this repo.

<ipython-input-20-c6dfcddf42b7> in <module>
----> 1 zfile = EosdisStore(netcdf, dmr_url=dmrpp)

~/miniconda3/envs/eisfire/lib/python3.9/site-packages/eosdis_store/stores.py in __init__(self, data_url, dmr_url)
    256         dmrpp = requests.get(dmr_url).text
    257         tree = ElementTree.fromstring(dmrpp)
--> 258         meta_store = to_zarr(tree)
    259         super(EosdisStore, self).__init__(meta_store, data_url)

~/miniconda3/envs/eisfire/lib/python3.9/site-packages/eosdis_store/dmrpp.py in to_zarr(root)
    277     zarr = {}
    278     dims = get_dimensions(root)
--> 279     zarr = group_to_zarr(root, dims)
    280     return zarr
    281

~/miniconda3/envs/eisfire/lib/python3.9/site-packages/eosdis_store/dmrpp.py in group_to_zarr(node, dims, prefix)
    245         # if this is an array, convert to zarr array
    246         if tag in TYPE_INFO:
--> 247             zarr_array = array_to_zarr(child, dims, prefix=prefix)
    248             zarr.update(zarr_array)
    249         # otherwise, if this is group or a Container Attribute - this has not been tested

~/miniconda3/envs/eisfire/lib/python3.9/site-packages/eosdis_store/dmrpp.py in array_to_zarr(node, dims, prefix)
    204                 zarray['compressor'] = None
    205             else:
--> 206                 raise Exception('Unrecognized compressionType: ' + compression)
    207             chunks = chunks_to_zarr(child)
    208             zarray.update(chunks['zarray'])

Exception: Unrecognized compressionType: deflate shuffle

This is a great package and is used in some of the NASA tutorials, but it has not been updated for awhile and when I install with conda it is downgrading my xarray package. This causes problems. Can you update to depend on a newer xarray?

I'm interested in generating DMR++ files for some HDF5 files, is there documentation on how to do this?