nasa-gibs / mrf Goto Github PK
View Code? Open in Web Editor NEWGDAL-compatible file format driver designed for fast access to imagery
License: Apache License 2.0
GDAL-compatible file format driver designed for fast access to imagery
License: Apache License 2.0
After "make clean" in the mrf folder, a gdal make will fail since it doesn't seem to depend on libLERC objects. At least in the GNUMakefile (*nix).
Deflate at level 0 is simply storing data, it should be disabled in MRF.
If no Write() is issued after Create(), the mrf files are not created.
GDAL PAM handles the stats (min, max, histogram ....)
Once calculated ones, for a version or a slice, the values will be reported for any other version or slice.
This is because GDAL is not aware of the versions or slices.
Can't easily be fixed, because it would require disabling PAM and implementing it inside MRF.
To generate correct stats for a different slice or version, remove the aux.xml file and calculate stats for the desired slice or version.
The output PNG can be considerably larger than input. Currently the code adds 100 bytes, it is not sufficient in case the data is random, or when tile is tiny and PPNG is used.
Valid quality values are between 0-99, while in gdal QUALITY=100 is very common.
Looks like GDAL does support palette with more than 256 entries for unsigned short int.
When such a file is copied into MRF, the resulting XML has more than 256 entries in the palette and fails to open.
The current behavior is that a zero on any band on input will result on zeros on all bands on output, since there is only one zero mask. This leads to some saturated colors pixels in input to be converted to black.
Instead, the default behavior should be to only flag as black pixels that have at least one zero value in a band and relatively small values in the other bands (very dark pixels). This would work better for most cases.
The original behavior (any zero masks the whole pixel) could be removed or made available via a JPEG specific create option.
I had genarate 4 bands mrf file;
i read every tile from .idx ,but how get the all bands data every tile?
It is part of GDAL at version 2.3, and supposed to be faster/better than DEFLATE
GDAL supports datamasks but MRF does not.
The MRF data file may accumulate unused space when using mrf_insert or when the multi-process safe is used. An utility to copy only the useful parts of the data file is needed, utility which should also preserve the sparse index file.
With some forethought environment variable(s) could control the location of the large files, without having to change the content of the metadata file itself.
I have tried the tool of optimizeraster and I would be able to convert landsat to mrf.
My current customer's demand is to convert from imagery of img.
look forward
Alison
This should improve the DEFLATE compression.
MRF has two internal overview resampling modes, one based on averaging and one based on nearest neighbor, both implemented only for overview factor of 2.
There are at least two problems with the current state:
Proposed solution:
There appears to be some resampling differences along border areas with nodata at lower resolution overviews (see sst_before/after.png)
Example GDAL commands:
gdalbuildvrt -q -te -180 -90 180 90 -vrtnodata 0 -input_file_list
gdal_translate -of MRF -co COMPRESS=PPNG -co BLOCKSIZE=512 -outsize 40960 20480
gdaladdo -q -r nearest sst.mrf 2 4 8 16 32 64 128
Copying MRFs is tricky, especially the ones with very large bounding boxes.
The canned index format helps, but it has to be uncanned before use.
It should be possible to make the canned index an alternative to the array version, which would make the MRF easier to transfer.
A side-effect will be that the resulting MRF will be read-only. Another side-effect is that cloning of such a file gets harder.
A related feature would be to combine all three components of an MRF into a single one, also enhancing transferabillity.
MRF currently supports mask via PAM, as external files. The external mask doesn't work through caching or cloning MRFs, it is ignored.
MRF should support a mask internal to the tile, at least for the JPEG compression format, which works through caching or cloning MRFs.
Since PNG requires big endian input data, the byte order of the input data is usually flipped on write. This is fine in most cases, but when the tile data is not discarted, it will have the bytes flipped. This happens for caching MRFs with PNG as the caching format.
As discussed, this project will no longer contain the MRF driver proper, which is maintained under the OSGeo/gdal project.
When copying from a VRT, the blocksize is inherited, which leads to tiny block size in MRF, it can be inefficient and leads to large index files.
The default size of 512x512 should be set when the block size is not explicitly set and the input block size is small.
Under version 3.2 of gdal, we were running a program to build mrf files using the following commands
self.outRaster = self.mrfDriver.Create( mrfFileName, self.config["numCols"], self.config["numRows"], 1, # number of bands, R G B self.config["dataType"], options=[ "Zsize={:d}".format(zSize), "UNIFORM_SCALE=2", # space for overviews self.config["MRF-OPTIONS"], "COMPRESS=LERC", "BlockSize={}".format(self.config["TILESIZE"]),
*the MRF-OPTIONS from our config file are "OPTIONS=V2=ON LERC_PREC=0.0001"
We recently updated to gdal 3.4.2 and have consistently received the error:
`Traceback (most recent call last):
File "/opt/anaconda3/envs/gdalenv/lib/python3.9/multiprocessing/pool.py", lin
e 125, in worker
result = (True, func(*args, **kwds))
File "/opt/anaconda3/envs/gdalenv/lib/python3.9/multiprocessing/pool.py", lin
e 51, in starmapstar
return list(itertools.starmap(args[0], args[1]))
File "/home/arcgis/gis-project/src/master.py", line 65, in execute_task
gdalMrf.driver(dtg, doy)
File "/home/arcgis/gis-project/src/mrf.py", line 468, in driver
self.run_day(mrfDataPath, year, doy)
File "/home/arcgis/gis-project/src/mrf.py", line 347, in run_day
self.mrf_3d_files(mrfDataPath, sliceNumber, year, doy)
File "/home/arcgis/gis-project/src/mrf.py", line 253, in mrf_3d_files
self.create_mrf(
File "/home/arcgis/gis-project/src/mrf.py", line 128, in create_mrf
self.outRaster = self.mrfDriver.Create(
File "/opt/anaconda3/envs/gdalenv/lib/python3.9/site-packages/osgeo/gdal.py",
line 2001, in Create
return _gdal.Driver_Create(self, *args, **kwargs)
RuntimeError: GDAL MRF: Error setting compression
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/arcgis/gis-project/src/master.py", line 150, in
main(params)
File "/home/arcgis/gis-project/src/master.py", line 95, in main
res = pool.starmap(execute_task, inputData)
File "/opt/anaconda3/envs/gdalenv/lib/python3.9/multiprocessing/pool.py", lin
e 372, in starmap
return self._map_async(func, iterable, starmapstar, chunksize).get()
File "/opt/anaconda3/envs/gdalenv/lib/python3.9/multiprocessing/pool.py", lin
e 771, in get
raise self._value
RuntimeError: GDAL MRF: Error setting compression`
Any assistance in enabling LERC compression under the new version would be appreciated. Let me know if more information or explanation is required.
Should test band by band with the correct NDV value. Currently it tests all bands against the first band NDV, which in rare cases is incorrect.
A global MRF file was generated from JPEG tiles covering only a small portion of the whole area, thereby leaving lots of blank space. The imagery returned does not look good.
Example GDAL commands:
gdalbuildvrt -q -te -180 -90 180 90 -vrtnodata 0 -input_file_list
gdal_translate -q -of MRF -co COMPRESS=JPEG -co BLOCKSIZE=512 -outsize 163840 81920
gdaladdo -q -r nearest test.mrf 2 4 8 16 32 64 128 256 512
An issue was found where the MRF index size apparently resets to 0 bytes. Given a large 15m product, the z-size was set to 163 levels, and the index looks okay:
gdal_translate -of MRF -co COMPRESS=PNG -co BLOCKSIZE=512 -outsize 2621440 1310720 -co ZSIZE=163 -co NOCOPY=true -co UNIFORM_SCALE=2 test.tif test.mrf
test.idx = 45578122592 bytes
When we try the product again with 164 levels, the index file gets created with 0 bytes:
gdal_translate -of MRF -co COMPRESS=PNG -co BLOCKSIZE=512 -outsize 2621440 1310720 -co ZSIZE=164 -co NOCOPY=true -co UNIFORM_SCALE=2 test.tif test.mrf
test.idx = 0 bytes
If I reduce the outsize, I can increase the z-size proportionally. However, the index appears to reach around 43GB before resetting to 0 bytes.
When opening an MRF using the :MRF:L name, the pixel size is wrong. It looks like it is off by one level, since L0 has the same pixel size as the base resolution.
File : mrf/src/gdal_mrf/frmts/mrf/mrf_band.cpp
fatal error: ../zlib/zlib.h: No such file or directory
In order to fix i've to modify +#include "zlib.h"
OR
You +#include <zlib.h>
either works.
GDAL_DISABLE_READDIR_ON_OPEN has two states:
FALSE (default), sidecar files are sought
EMPTY_DIR - Disable looking for sidecar files
However, with MRF, some files are still sought. This takes considerable time in a busy folder or virtual storage (VSIL)
FALSE: 7 files
*.mrf.aux.xml, *.aux, *.AUX, *.mrf.aux, *.mrf.AUX, *.mrf.msk, *.mrf.MSK
EMPTY_DIR, it is still looking for 3 files:
*.mrf.aux.xml, *.mrf.msk, *.mrf.MSK
Even Roualt has done all the work for integration in GDAL 2.0, I've applied his patches one by one in the lp branch, with references to the GDAL trac, ticket 6342.
Most changes were cosmetic or minor, yet there were a couple of potential memory leaks and issues.
LERC format code is the most affected, because it made heavy use of unaligned memory reads/writes. These are not an issue on Intel x86 and x64 CPUs and don't reduce performance on recent CPUs either. But code checking tools warn/complain about the practice.
The expedient solution makes heavy use of memcpy, which probably reduces performance. Maybe a better solution can be found
Maybe insert into the using the file.mrf:MRF:L[0-9]* notation or some other command line flag supporting a direct insert of input into a specific level in an existing MRF, with no overviews as a combined option.
For large files, the jxl bulk conversion still takes a very long time, it could easily be done in parallel
Skipping level 1 (or every other level) would save significant space without incurring much of a performance penalty when used within gdal. Implementing this feature across all supported tile formats would save a significant amount of space.
For best performance, tiles should be stored in 2x2 groups, including the base level
LERC is a stand-alone project in github.com/esri/lerc.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.