Comments (1)
My understanding of GRIB2 is that it's based on quantizing floating-point data to integers (following quantization levels determined to be appropriate for each physical quantity) and then applying JPEG2000, PNG, or other standard image compression algorithms to 2D slices. Both JPEG2000 and PNG are limited in the precision and data dimensionality they support. The JPEG2000 spec does have extensions for compressing 3D data, though as far as I know few tools support Part 10 of the spec, and there's to my knowledge no support for 3D + time. In terms of error tolerances, GRIB2 supports bounded errors only when used in conjunction with lossless compression. Finally, serial zfp is 1-2 orders of magnitude faster than JPEG2000 and also has support for parallel compression and other features, like random access.
That said, there's no reason why one could not make such a comparison. Indeed, we do include one comparison with JPEG2000 on the zfp homepage based on the implementation of Woodring et al. for compressing climate data using JPEG2000.
I also do work closely with climate scientists at NCAR who favor zfp and fpzip over GRIB2. See this paper and references therein on further discussion of and comparisons with GRIB2.
Finally, the obvious "hybrid" would be to apply zfp to quantized GRIB integer data. However, zfp is not all that well suited for lossless compression (fpzip performs far better). Note that the first stage of zfp compression (the block-floating-point transform) essentially performs uniform quantization (in fixed-accuracy mode). In conjunction with the other zfp steps, this quantization allows bounding the error and exploiting higher-dimensional correlations.
A perhaps better compromise is to use zfp with HDF5 via the H5Z-ZFP plugin.
from zfp.
Related Issues (20)
- Python binding installation fails HOT 18
- ppc32: index.hpp: error: integer constant is too large for ‘unsigned long’ type HOT 3
- How to run zfp as a c code with gcc HOT 6
- Is psnr formula error in ZFP? HOT 2
- Missing `void` in functions that take no parameters
- Tiny but above subnormal numbers not handled correctly
- Numpy 2.0 HOT 11
- Cuda mode ZFP examples HOT 4
- Compilation issues with zfp HOT 2
- Running issues with zfp and cuda HOT 3
- Errors reported with scikit-build when running CMake with BUILD_ZFPY enabled HOT 1
- Test failures and segfaults when building with Visual Studio 2022 compiler v14.38.33130 HOT 13
- 2D Point data curiosly compresses way better at 1D HOT 4
- Deserializing const_array HOT 2
- Issue with running the diffusion program in the examples folder HOT 3
- Why strides are ignored in zfp_field_metadata()? HOT 4
- Syntax error in setup.py HOT 1
- Adjusting test cases from mock tests to proper tests for `testZfpCuda4d<datatype>` HOT 3
- example array.cpp error:C4146 Unary minus operator applied to unsigned type, the result is still unsigned. store.hpp 241 HOT 3
- The data is wrong after decompression and decompression by zfp_field_1d, but it is correct when I use zfp_field_2d HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from zfp.