Chunking support about asdf HOT 2 OPEN

Thanks for opening this issue.

There has been some work adding support for the zarr storage format within ASDF. This is implemented via an extension: https://github.com/asdf-format/asdf-zarr It's a new package so please let me know if it's something you plan to use "in production" (so we can give it another review, also feel free to give it a try and open issues if you find anything). The extension offers a few options:

• storing the zarr data inside ASDF blocks (with a chunk per block, I think most similar to what you described)
• referencing external zarr storage (either DirectoryStore "flat files", S3 stores, or any of the many formats zarr supports).

The use of zarr also opens up a second place where compression can be controlled (which can get a bit confusing).

from asdf.

@braingram Nice! We are currently discussing storage formats, and both ASDF and Zarr are contenders that have various advantages and disadvantages. On the surface, using Zarr chunking with ASDF single-file storage seems like an excellent choice. I will have a look.

from asdf.