Persisting ZFP compressed arrays about zfp HOT 9 CLOSED

Mark, I think the issue of storing a compressed array to file is a little out of scope for the zfp API itself. The 0.5.5 release will have support for serializing a compressed array to memory and a factory function for constructing an array from an in-memory compressed buffer (see https://github.com/LLNL/zfp/tree/feature/compressed-array-header). These functions use zfp's header to fully describe array rank, dimensions, scalar type, compression mode, etc., and will allow you, for instance, to write out a compressed array and later decompress it using the command-line tool, as if the compressed file had been generated by the command-line tool in fixed-rate mode.

As your code example illustrates, zfp already has a method that return a pointer to the compressed byte stream (array::compressed_data()) that the caller can use to write the compressed data to disk (or read compressed data into). And as you point out, there are many ways of writing data to disk, e.g., via fwrite, HDF5, ADIOS, or other container formats or I/O interfaces. You might also want to communicate arrays to other processors, e.g., via MPI, over a socket, over a named pipe, etc. I think it's best to leave this out of the zfp API while providing the functionality necessary to construct and serialize arrays.

If you want to have a generic interface for doing this, then I don't think C-like function pointers are the right approach. The natural C++ way of doing this would be via inheritance, possibly using virtual functions. Though I'm not convinced these need to be member functions.

With respect to HDF5, I'm not sure I understand what problem these two functions address. Can you please elaborate?

from zfp.

Regarding other forms of writing (or reading) a ZFP compressed array...I agree.

You don't want to complicate ZFP implementation with this or add depdendencies on other packages people might serialize ZFP arrays through (e.g. MPI, HDF5, etc). I get that. I agree.

But, without providing some abstraction in ZFP itself (you don't have to call it load or store, call it stream_out and stream_in or something more general), everyone is forced to inheret to embue any ZFP instance with this ability. I know that is the C++ way of doing this kind of thing. I just view it as common use case that is then made more complicated for many consumers of the ZFP compressed array classes interface for lack of some abstraction in ZFP itself to support/facilitate it.

Case in point, don't a lot of the ZFP array classes example codes in ZFP write ZFP arrays to files. Seems like a common use case.

My reason for suggesting a static approach is to make it easy, a single function call, to ensure any ZFP array class instance existing or created in the future, is them embued with this ability.

Maybe its worth considering the following use case....Suppose I am using ZFP array classes throughout my code. Now, I want to save them to disk, or MPI send them to different ranks, etc. What is the coding involved to do that given how ZFP array classes work now and is there anything ZFP array classes can do to simplify that. I think there is. But, I don't think it comes for free. I think ZFP array classes then need some kind of a supporting abstraction.

Thats all I am saying.

from zfp.

Currently we do not have a code example that writes zfp arrays to disk. Nor am I aware of an application that does this, although I do expect this to become a more common use case in the future as apps start adopting zfp arrays in their computations. This is one reason why we're adding a mechanism to Conduit for expressing zfp arrays as Conduit nodes.

What I am still unsure about is what your proposed two function pointers accomplish that cannot already be done as easily through other means. Note that the load & store functions still have to be implemented by the application, and will likely require additional arguments (e.g., file descriptor) as source/target that are not naturally handled through a fixed function prototype.

Here are a few alternatives I see. (1-3) are already available.

1. Use compressed_data() and compressed_size() in application-defined load/store functions to read/write the compressed data, as in the example you provided. Metadata can either be stored by the application verbatim or via zfp's header. By accessing the compressed data through a pointer, this avoids copying any data.

2. Use the soon to be released factory function to load (construct) a zfp array from memory and a combination of write_header() and compressed_data() to store. The caller would either fwrite() the header and compressed data directly or concatenate them to a user-allocated memory buffer before outputting. One disadvantage is that this makes a separate copy of the compressed data before doing I/O. I agree that this could be made simpler. @salasoom, do we not have a single function that serializes the array as an inverse of the factory function?

3. Derive a new class from a zfp array that adds load & store functions. The programming effort is essentially identical to your proposal of adding function pointers to the zfp array classes. A little less, actually, since you do not need to set function pointers and define the functions and their prototypes externally. One benefit of this approach is that load & store could take any number of other arguments of any type needed for I/O.

4. Add virtual load & store functions to zfp's classes, making these officially part of the zfp API. The programming effort is the same as in the previous solution, although this approach would officially extend the zfp API. One benefit over (3) is a more uniform API; one drawback is that other arguments needed for I/O must be passed through a common API (or encapsulated in the derived class itself).

Your proposed solution differs from these, and I'd like to understand better its advantages of these alternatives.

from zfp.

What about mmap-backed ZFP compressed arrays? I ended up modifying array/zfparray.h and array/zfparray3.h after importing the whole zfp-0.5.4 into own github project:

https://github.com/jvo203/FITSWebQL

The 3D cubes that need to be handled in the near future are astronomical (they come from radio-astronomy and their sizes are "astronomical"): over 256TB uncompressed per cluster node. There is no way ZFP-compressed arrays can all fit in RAM. Also, the data contains NaN so there is a need to substitute 0.0 for NaNs and track non-values via separate compressed bit arrays.

from zfp.

@jvo203 We will not have mmap support right away. The main issue is dealing with ownership of memory. Currently zfp arrays own their own memory, and our (de)serialization API will make a deep copy in case the user allocates data for compressed storage themselves (as would be needed with mmap). We're considering adding support for shallow copies by maintaining who owns the relevant memory buffer, e.g., so that zfp does not attempt to deallocate user-allocated memory when arrays are destructed or resized.

As for NaNs, this is something that we've been working toward for quite some time. We don't have a solution yet that is ready for release, but we will handle NaNs in the future. I will caution that replacing NaNs with zeros will in most cases introduce artificial discontinuities that hurt compression. zfp has to deal with the same problem for arrays whose dimensions are not multiples of four, in which case it inserts padding for partially filled blocks that promotes compression. We ought to include a utility function that does something similar for more general cases of partially filled blocks, but I doubt this could be accomplished before the next release. If this is a problem for you, please file a separate issue and I'll see if we can provide such a function on a feature branch after we're done with the 0.5.5 release.

from zfp.

Thank you for a prompt response. Regarding NaNs, what would be the recommended optimum replacement value to put in partially-filled blocks? The arithmetic mean of all the other non-NaN values? We also have blocks that contain nothing but NaNs, in which case I guess 0.0 would be OK.

Checking for NaN and putting 0.0 is done in a separate SIMD code that goes through each widthxheight 2D plane at once (it also converts from big-endian to little-endian) without considerations for 4x4 block boundaries. As this is ultimately a 3D cube, checking each 4x4x4 block for NaN and calculating block-specific optimum replacement value prior to pushing values to a mutable private_view (OpenMP multithreading operating on widthxdepthx4 private views) seems a bit burdensome.

Typically NaNs appear at the boundary of the data so I guess we can live with sub-optimum compression here and there, as long as it is not pervasive.

from zfp.

Thank you for a prompt response. Regarding NaNs, what would be the recommended optimum replacement value to put in partially-filled blocks? The arithmetic mean of all the other non-NaN values?

Unfortunately, the optimal padding is not a single fill value but depends on the spatial configuration of available values. For partial blocks, zfp uses one of the up to three available values within the same block of four values along one dimension. For optimal compression, a better solution would be to perform a polynomial fit, but that can result in overshoots that lead to overflow.

We also have blocks that contain nothing but NaNs, in which case I guess 0.0 would be OK.

Yes, that would be ideal.

Checking for NaN and putting 0.0 is done in a separate SIMD code that goes through each widthxheight 2D plane at once (it also converts from big-endian to little-endian) without considerations for 4x4 block boundaries. As this is ultimately a 3D cube, checking each 4x4x4 block for NaN and calculating block-specific optimum replacement value prior to pushing values to a mutable private_view (OpenMP multithreading operating on widthxdepthx4 private views) seems a bit burdensome.

It may not be all that bad. If you're using AVX for this, then you're processing groups of four values anyway. You'd want to use zfp's iterators to initialize one block at a time. Unfortunately zfp's views do not yet have iterators, which is something that will be addressed in the coming year.

Can this preprocessing be done to the uncompressed data before "converting" it to zfp?

What is the third dimension in this case? Color/spectral band? If so, then there may not necessarily be any correlation across bands, in which case you might favor 4 2D arrays.

Typically NaNs appear at the boundary of the data so I guess we can live with sub-optimum compression here and there, as long as it is not pervasive.

In the case of zfp's compressed arrays, the compression ratio is fixed, so what would suffer is accuracy rather than compression. You'd most certainly be better off replacing NaNs with some kind of average rather than zeros, in case you're looking for a simple solution.

from zfp.

Yes, the pre-processing part (endianness, NaNs, non-value bit masks) only needs to be done once, upon the first-time reading of an astronomical FITS file containing a 3D data cube. Afterwards, all the subsequent data accesses only read the FITS file header (skipping the FITS data cube) and go directly to an offline already-compressed ZFP cube (mmapped file).

Radio-astronomy FITS data cubes: width x height x depth x polarisation . I am not handling the polarisation part for now, it will come later. The depth dimension is frequency bands. So we are dealing with 2D planes for a number of frequency bands, with the pixels containing intensity values [Jy/beam]. Yes, in most cases there is a fair amount of correlation between neighbouring 2D planes, hence there is a single 3D ZFP cube instead of a collection of separate 2D ZFP planes.

from zfp.

@markcmiller86, @jvo203: I wanted to let you know that we have released zfp 0.5.5, which adds support for (de)serializing compressed arrays. This is actually quite straightforward (see the examples here for how to do this using fread and fwrite).

The API does not yet support mmap, for the reasons already discussed. Doing so would require a more general mechanism for tracking who owns the compressed data buffer. I do not foresee any particular challenges with supporting this, but we just didn't have time to add such functionality to this release.

@jvo203, since this issue has more to do with managing memory ownership (an issue not limited to mmap) and less to do with (de)serialization, I would prefer if you'd submit a separate issue that more directly addresses your request so we can close the issue opened by @markcmiller86. We should be able to support mmap in the next release.

from zfp.