Giter VIP home page Giter VIP logo

mrf's People

Contributors

alexanderbock avatar bbradbury avatar jdrodjpl avatar jeffreyrhall avatar jtroberts avatar lucianpls avatar mcechini avatar mike-mcgann avatar rouault avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mrf's Issues

For 3-rd dimension or versioned MRFs, the statistics are not reliable

GDAL PAM handles the stats (min, max, histogram ....)
Once calculated ones, for a version or a slice, the values will be reported for any other version or slice.
This is because GDAL is not aware of the versions or slices.
Can't easily be fixed, because it would require disabling PAM and implementing it inside MRF.
To generate correct stats for a different slice or version, remove the aux.xml file and calculate stats for the desired slice or version.

PNG output buffer size insufficient

The output PNG can be considerably larger than input. Currently the code adds 100 bytes, it is not sufficient in case the data is random, or when tile is tiny and PPNG is used.

Palette of more than 256 entries is not supported

Looks like GDAL does support palette with more than 256 entries for unsigned short int.
When such a file is copied into MRF, the resulting XML has more than 256 entries in the palette and fails to open.

Make the JPEG/Zen mask tolerant of pure colors in band interleaved MRFs

The current behavior is that a zero on any band on input will result on zeros on all bands on output, since there is only one zero mask. This leads to some saturated colors pixels in input to be converted to black.
Instead, the default behavior should be to only flag as black pixels that have at least one zero value in a band and relatively small values in the other bands (very dark pixels). This would work better for most cases.
The original behavior (any zero masks the whole pixel) could be removed or made available via a JPEG specific create option.

mrf_compact

The MRF data file may accumulate unused space when using mrf_insert or when the multi-process safe is used. An utility to copy only the useful parts of the data file is needed, utility which should also preserve the sparse index file.

how to convert imagery format of img to mrf

I have tried the tool of optimizeraster and I would be able to convert landsat to mrf.
My current customer's demand is to convert from imagery of img.
look forward
Alison

Internal resampling algorithms

MRF has two internal overview resampling modes, one based on averaging and one based on nearest neighbor, both implemented only for overview factor of 2.
There are at least two problems with the current state:

  • The code does not check that the factor is indeed 2. This has led to the gdal unit testing the internal modes with a factor of 3. Which works, but produces bad results. It passes the numeric test, but the result is not valid.
  • The name for the nearest neighbor resampling mode is badly chosen. GDAL accepts any name that starts with "near" as it's version. The MRF tests first, and currently uses the same test, so the user can't really tell which algorithm is used.

Proposed solution:

  • Rename the nearest neighbor resampling to "nnb". This will break backwards compatibility, some scripts and potentially some codes will need to change.
  • Reject unsupported overview scaling factors. This will not cause serious disruptions, as the result would have been wrong anyhow.

Resampling differences along border areas

There appears to be some resampling differences along border areas with nodata at lower resolution overviews (see sst_before/after.png)

sst_before
sst_after

Example GDAL commands:
gdalbuildvrt -q -te -180 -90 180 90 -vrtnodata 0 -input_file_list
gdal_translate -of MRF -co COMPRESS=PPNG -co BLOCKSIZE=512 -outsize 40960 20480
gdaladdo -q -r nearest sst.mrf 2 4 8 16 32 64 128

Support single file MRFs, with compact index

Copying MRFs is tricky, especially the ones with very large bounding boxes.
The canned index format helps, but it has to be uncanned before use.
It should be possible to make the canned index an alternative to the array version, which would make the MRF easier to transfer.
A side-effect will be that the resulting MRF will be read-only. Another side-effect is that cloning of such a file gets harder.
A related feature would be to combine all three components of an MRF into a single one, also enhancing transferabillity.

Internal mask support

MRF currently supports mask via PAM, as external files. The external mask doesn't work through caching or cloning MRFs, it is ignored.
MRF should support a mask internal to the tile, at least for the JPEG compression format, which works through caching or cloning MRFs.

PNG caching of 16bit data flips endianess

Since PNG requires big endian input data, the byte order of the input data is usually flipped on write. This is fine in most cases, but when the tile data is not discarted, it will have the bytes flipped. This happens for caching MRFs with PNG as the caching format.

small block size when copying via virtual dataset

When copying from a VRT, the blocksize is inherited, which leads to tiny block size in MRF, it can be inefficient and leads to large index files.
The default size of 512x512 should be set when the block size is not explicitly set and the input block size is small.

LERC compression unknown with 3.4.2

Under version 3.2 of gdal, we were running a program to build mrf files using the following commands
self.outRaster = self.mrfDriver.Create( mrfFileName, self.config["numCols"], self.config["numRows"], 1, # number of bands, R G B self.config["dataType"], options=[ "Zsize={:d}".format(zSize), "UNIFORM_SCALE=2", # space for overviews self.config["MRF-OPTIONS"], "COMPRESS=LERC", "BlockSize={}".format(self.config["TILESIZE"]),
*the MRF-OPTIONS from our config file are "OPTIONS=V2=ON LERC_PREC=0.0001"
We recently updated to gdal 3.4.2 and have consistently received the error:

`Traceback (most recent call last):
File "/opt/anaconda3/envs/gdalenv/lib/python3.9/multiprocessing/pool.py", lin
e 125, in worker
result = (True, func(*args, **kwds))
File "/opt/anaconda3/envs/gdalenv/lib/python3.9/multiprocessing/pool.py", lin
e 51, in starmapstar
return list(itertools.starmap(args[0], args[1]))
File "/home/arcgis/gis-project/src/master.py", line 65, in execute_task
gdalMrf.driver(dtg, doy)
File "/home/arcgis/gis-project/src/mrf.py", line 468, in driver
self.run_day(mrfDataPath, year, doy)
File "/home/arcgis/gis-project/src/mrf.py", line 347, in run_day
self.mrf_3d_files(mrfDataPath, sliceNumber, year, doy)
File "/home/arcgis/gis-project/src/mrf.py", line 253, in mrf_3d_files
self.create_mrf(
File "/home/arcgis/gis-project/src/mrf.py", line 128, in create_mrf
self.outRaster = self.mrfDriver.Create(
File "/opt/anaconda3/envs/gdalenv/lib/python3.9/site-packages/osgeo/gdal.py",
line 2001, in Create
return _gdal.Driver_Create(self, *args, **kwargs)
RuntimeError: GDAL MRF: Error setting compression
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/arcgis/gis-project/src/master.py", line 150, in
main(params)
File "/home/arcgis/gis-project/src/master.py", line 95, in main
res = pool.starmap(execute_task, inputData)
File "/opt/anaconda3/envs/gdalenv/lib/python3.9/multiprocessing/pool.py", lin
e 372, in starmap
return self._map_async(func, iterable, starmapstar, chunksize).get()
File "/opt/anaconda3/envs/gdalenv/lib/python3.9/multiprocessing/pool.py", lin
e 771, in get
raise self._value
RuntimeError: GDAL MRF: Error setting compression`

Any assistance in enabling LERC compression under the new version would be appreciated. Let me know if more information or explanation is required.

MRF problems in JPG with lots of blank space

A global MRF file was generated from JPEG tiles covering only a small portion of the whole area, thereby leaving lots of blank space. The imagery returned does not look good.

modis_err

Example GDAL commands:
gdalbuildvrt -q -te -180 -90 180 90 -vrtnodata 0 -input_file_list
gdal_translate -q -of MRF -co COMPRESS=JPEG -co BLOCKSIZE=512 -outsize 163840 81920
gdaladdo -q -r nearest test.mrf 2 4 8 16 32 64 128 256 512

MRF index size issue with z-dimension

An issue was found where the MRF index size apparently resets to 0 bytes. Given a large 15m product, the z-size was set to 163 levels, and the index looks okay:

gdal_translate -of MRF -co COMPRESS=PNG -co BLOCKSIZE=512 -outsize 2621440 1310720 -co ZSIZE=163 -co NOCOPY=true -co UNIFORM_SCALE=2 test.tif test.mrf
test.idx = 45578122592 bytes

When we try the product again with 164 levels, the index file gets created with 0 bytes:

gdal_translate -of MRF -co COMPRESS=PNG -co BLOCKSIZE=512 -outsize 2621440 1310720 -co ZSIZE=164 -co NOCOPY=true -co UNIFORM_SCALE=2 test.tif test.mrf
test.idx = 0 bytes

If I reduce the outsize, I can increase the z-size proportionally. However, the index appears to reach around 43GB before resetting to 0 bytes.

Broken when tyring to build on RHEL 7

File : mrf/src/gdal_mrf/frmts/mrf/mrf_band.cpp
fatal error: ../zlib/zlib.h: No such file or directory

include "../zlib/zlib.h"

In order to fix i've to modify +#include "zlib.h"
OR
You +#include <zlib.h>
either works.

sidecar files are sought despide the GDAL_DISABLE_READDIR_ON_OPEN

GDAL_DISABLE_READDIR_ON_OPEN has two states:
FALSE (default), sidecar files are sought
EMPTY_DIR - Disable looking for sidecar files
However, with MRF, some files are still sought. This takes considerable time in a busy folder or virtual storage (VSIL)

FALSE: 7 files
*.mrf.aux.xml, *.aux, *.AUX, *.mrf.aux, *.mrf.AUX, *.mrf.msk, *.mrf.MSK
EMPTY_DIR, it is still looking for 3 files:
*.mrf.aux.xml, *.mrf.msk, *.mrf.MSK

GDAL 2.0 integration

Even Roualt has done all the work for integration in GDAL 2.0, I've applied his patches one by one in the lp branch, with references to the GDAL trac, ticket 6342.

Most changes were cosmetic or minor, yet there were a couple of potential memory leaks and issues.

LERC format code is the most affected, because it made heavy use of unaligned memory reads/writes. These are not an issue on Intel x86 and x64 CPUs and don't reduce performance on recent CPUs either. But code checking tools warn/complain about the practice.
The expedient solution makes heavy use of memcpy, which probably reduces performance. Maybe a better solution can be found

Add parallel jxl conversion

For large files, the jxl bulk conversion still takes a very long time, it could easily be done in parallel

Add overview mode with missing level 1

Skipping level 1 (or every other level) would save significant space without incurring much of a performance penalty when used within gdal. Implementing this feature across all supported tile formats would save a significant amount of space.

  • Add a new overview mode
  • Modify tile read to read 4 tiles from lower level and downsample to generate a top tile, using avg or nnb mode
  • Modify CreateOverviews to build overviews

For best performance, tiles should be stored in 2x2 groups, including the base level

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.