"zero_is_not_bad" hint implemented for RW2, never active about rawspeed HOT 23 OPEN

Let me summarize my position/view:

• Based on data I see in frames provided above,
  only considering pixels [that are zero] to be bad may not be enough,
  there are some pixels that are slightly above zero.
• If that threshold (if the pixel value is less than, or equal to the value, then it is bad)
  is somehow stored in the makernotes/exif, then we can fix that in rawspeed.
• If the badpixels map is somehow stored in the makernotes/exif, then we can fix that in rawspeed.
• Else, i don't think we can/should fix this in rawspeed.
• darktable coldpixels module is a good idea regardless of the resolution of this particular problem.

from rawspeed.

The wonders of R seem to provide additional insight, see script below.

The only data point I have investigated with that script is the one high-resolution file that is also on RUP.

What the script seems to indicate is that, yes, there are zero values in the PPM file (which is dumped straight from the decoded values). But the script claims that all these values are within a well-defined rectangle (bounding box):

[10403, 10447] x [12, 7790]

within that binary decoded matrix. rawspeed sees the following metadata:

dimUncropped: 10480x7794
dimCropped: 10368x7776
cropOffset: 18x10

And that means that the cropped area == "bounding box within the binary dump" as seen by rawspeed is

[18, 10386] x [10, 7786]

And, unless it is too late, and I am too tired, this "crop data bounding box" does not overlap with the "bad data bounding box". Which would be good.

Does that make sense?

R script - you may need to install the packages before use:

# pixmap library 0.4-11 cannot read two-byte (maxval = 65535) files
#    https://cran.r-project.org/package=pixmap
# A fixed version is required, with a patch similar to 
#    https://github.com/shoffmeister/pixmap/commit/2033b88bf0e77d7ac6ef56b82f7f2e791cbcbce3
library(pixmap)

library(spatstat)

highres_filename <- "github/rawspeed/Panasonic - DC-G9 - 4_3_real_HR.RW2.ppm"

highres_ppm <- read.pnm(file = highres_filename)
all_zero_coordinates <- which(highres_ppm@grey==0, arr.ind = TRUE, useNames = FALSE)
bb <- bounding.box.xy(all_zero_coordinates[,2], all_zero_coordinates[,1])

print("Panasonic G9 high-resolution raw files have a documented dimension of 10368x7776.")
print("The following window identifies the binary data coordinates where zero values are rendered:")

bb

from rawspeed.

For the record, I just took a photo with a Panasonic G9 in "real" high-resolution mode, in a pitch black environment, with electronic shutter at 1/32000. It (hopefully) does not get any more black than that.

Result: I get 1589 "instances" of zero in such a file (out of 81681120 pixels). Now, the "simulate as a single shot" high-resolution file (which is just the first exposure without any compositioning) has no instances of zero.

I don't know what to make of that.

There is a chance, that the Panasonic exposure assembly implementation (from 8 build 1) might, computationally, produce zero values. In that case, zero would really be zero and "not broken"?

from rawspeed.

@shoffmeister, could you please share some black frame images (with lens caps on) in high-res mode (different ISO, including some long exposure, like 20sec at highest possible ISO value)?

It is easy to estimate probability of 'natural zeroes' after estimating standard deviation of black frame files..

from rawspeed.

Proposal for test design:
ISO 200; 1/32000; f/22; electronic shutter
ISO 3200; 1 second; f/22; electronic shutter
ISO 25600; 60 seconds; f/22; mechanical shutter

Anything beyond that? I would only want to do that once, so prep once, shoot once :)

This would generate one "simulate single shot" (first exposure) and one real high-resolution shot (eight exposure composited into one) - 32 MB + 125 MB = 157 MB for each shot.

Any good place to store that?

from rawspeed.

Oh, and Panasonic Japan have announced a firmware update for the G9 (and GH5/GH5s) by end of May 2018, see https://news.panasonic.com/jp/topics/160585.html

For the G9 they promise updating the high-resolution mode: "Improvement of motion correction performance of high resolution mode."

It might be convenient to (also) look at the data after the firmware update?

from rawspeed.

@shoffmeister, for temp transfer we suggest our users to use WeTransfer service (free option). The link does not last long, but enough for transfers.

I'm interested to have 1 second and 60 second at ISO 25600 (to look at possible noise reduction signs)

from rawspeed.

Updated proposal for test design:
ISO 200; 1/32000; f/22; electronic shutter
ISO 3200; 1 second; f/22; electronic shutter
ISO 25600; 1 second; f/22; mechanical shutter
ISO 25600; 60 seconds; f/22; mechanical shutter

from rawspeed.

Ho-humm - my proposal is stupid - it does not cover high-resolution mode at all. High-resolution mode on the Panasonic G9 only works with the electronic shutter (ha!), and has an ISO cap at 1600, and an aperture cap (until the May 30 firmware update) at f/8.

Lets try this, then:
ISO 1600; 1 second; f/8; electronic shutter
ISO 200; 1 second; f/8; electronic shutter
ISO 200; 1/32000; f/8; electronic shutter
ISO 1600; 1/32000;f/8; electronic shutter

These exposures, in exactly that order, with "real high-resolution" and "simulated single shot" RW2s (in that order), are at https://www.dropbox.com/s/9ddkdi9jkp3tomo/darktable%20tests.zip?dl=0

from rawspeed.

Thank for test set.
Here is histogram of ISO200/1/32000 hi-res image (zero_is_black not used in LibRaw for these files):

(vertical scale is logariphmic).

Things to mention:

• at this low ISO/short exposure, zero pixels are not expected (see bell curve shape around black level), but exists (about 100 pixels in each channel).
• there are 'mirrored outliers' with similar (but not equal) counts in R and G2 channels (but not in G1/B). This is very likely, that these pixels should be masked too, but now way to do it on real image (very interesting too see, if this 'hotter' pixels position is fixed or not; if fixed, black frame subtraction will help).
• In Blue channels 2nd set of outliers lives at levels 9-10, not at ~260.

I do not see any outliers in single-frame shots (sure, I've turned off Panasonic bad pixels filtration in RawDigger vendor-specific settings).

Conclusion:

• zero pixels should be filtered out for G9/hires. May be, not only zero ones, but <15 or so (blue channel)
• other outliers needs that too, but no way to do that (unless these pixels coordinates are stored in makernotes and someone will hack this).

from rawspeed.

@LibRaw you can just copy-paste images into here, it will just work.

Detecting bad pixels by comparing them against some magic threshold (especially non-zero)
feels like a very dirty hack :/
If the bad pixel info is stored in makernotes, hell yes, let's use that.

Otherwise, i would personally prefer for a new darktable iop,
coldpixels, which would run before rawprepare (which does black level subtraction, b/w scaling)..

from rawspeed.

I think we are looking at very peculiar case here - this is a file that has been cobbled together by in-camera processing of eight exposures.

@LibRaw - what does the matching "simulated single shot" look like (P1001271.RW2 - 32 MB)? (Missed the fact that you had already replied - they have no zeros) This simulated single shot is, according to the Panasonic manual, the unprocessed first exposure. That plus the additional seven exposures then make up the final composited high-resolution raw. If my submission to RPU is anything to go by, I would expect no zero values. This then would imply that the compositing algorithms used by Panasonic may have a problem?

Now, widening the net, is there any knowledge / awareness what high-resolution files produced by the Olympus OMD E-M1 II look like with respect to compositing?

from rawspeed.

@LibRaw - would it provided any insight if I shot plain dark exposures at various ISO settings?

Like, really, plain exposure (i.e. not high-resolution) at
ISO 25600/1s/f22, mechanical shutter
ISO 25600/60s/f22, mechanical shutter

That will write in the "old" RW2 format, but might provide insight into that specific sensor that we are looking at?

from rawspeed.

@LebedevRI - perhaps a semi-generic pixel filter iop could be of benefit in darktable?

option: filter low values; low value threshold; substitute value
option: filter high values; high value threshold; substitute value

(semi-generic as the transformation function is just a step function)

In the case of the G9, I will be applying the firmware upgrade as soon as possible (very early June). Perhaps Panasonic have fixed / tweaked the algorithm used in compositing.

Getting to the general case of the "zero_is_not_bad" hint, right now it would seem as if

• implementation of the zero_is_not_bad hint in RW2 decoder can be removed
• decoded value is zero-processing should be added to PanasonicDecompressorV5 (this affects both the G9 high-resolution and the GH5s normal case)

?

from rawspeed.

Generic stuff usually does everything badly with bad performance.

option: filter high values; high value threshold; substitute value

There is already a hotpixels module (right before demosaic),
which does a fuzzy detection of hot pixels by comparing with surrounding pixels.
It won't really work well for these dark pixels, because even thought they are
initially dark (more or less below the black level), it runs after black level subtraction,
and after black level subtraction they are just zero.

So i think only the coldpixel side is missing.
I'd write it, but i kinda don't really feel like touching dt/c code.

Getting to the general case of the "zero_is_not_bad" hint, right now it would seem as if

• implementation of the zero_is_not_bad hint in RW2 decoder can be removed

I do agree that we could drop it, but i think it is semi-useful as a escape hatch for fuzzing.
So i'm not seeing much benefit in dropping it.

• decoded value is zero-processing should be added to PanasonicDecompressorV5 (this affects both the G9 high-resolution and the GH5s normal case)

I think more data points are needed. (+ see #137 (comment))
So far only one single raw (high-res, from not the latest camera) showed any differences.

from rawspeed.

Folks,

1. Based on data I see in frames provided above, I do not see any image-quality problems with filtering of non-zero (near-the-zero) pixels in blue channel: these pixels are way below possible level (compare bell-shape with values). If not fixed, these pixels may provide black dots on some even (bright) background.
   So, generic 'near_zero_is_bad' module looks as good idea.

2. The 'warm' pixels are another (bad) story: these pixels are above 7 stops above black level, so will be clearly visible as bright-enough dots on dark background (and demosaic will enlarge these points). Hope hotpixels module will fix it.

BTW, unless we do some scientific work (not the case for G9 camera), masking several 100s of outlier pixels (out of millions) will never degrade image quality unless these pixels are in some compact clusters. So, masking (e.g. averaging with surround 4 or so values) will not degrade visible image quality.

And, again, if these pixels are at fixed positions (I do not know, but may be Panasonic handles bad pixels map this way?), it is much better to mask it using camera-specific fixed mask, not by analysis of current frame.

from rawspeed.

Are there tools which would allow targetted visual inspection of the problematic pixels within an a larger image context?

Or just simply tools which would conveniently collect the set of "problematic" pixels?

With those tools, it could be possible to determine

• the stability of the problematic pixel coordinates given comparable light scenarions ("always all dark")
• the behaviour of the compositing algorithm when used on a wildly varying set of input files ("some proper exposure of something")

Hmm - rstest could serve as an inspiration as it is aware of "bad pixel tracking":

  APPEND(&oss, "badPixelPositions: ");
  {
    MutexLocker guard(&r->mBadPixelMutex);
    for (uint32 p : r->mBadPixelPositions)
      APPEND(&oss, "%d, ", p);
  }

That data on a scatter plot for each pixel (as decoded) might show something?

And that, then, at least allows building a meaningful hypothesis on whether the Panasonic high-resolution photo camera compositing algorithm is the root cause for the data, or whether it is the the sensor itself.

If it is the algorithm, one could try to bring this to Panasonic's attention, and perhaps Panasonic might fix it.

Now, for the GH5s - which is decoded by PanasonicDecompressorV5 - the zero story is unclear to me. There will be no compositing (the sensor is not stabilized, no high-resolution mode), so it is straight sensor data. And if the G9 "simulate single shot" data is to go by, then that data might have zero for pixels which are bad?

from rawspeed.

PS: Just noticed that only two raw format decoders have awareness of bad pixels: DNG, RW2 (except for the new v5 raw format).

So, doing statistics on many G9 high-resolution photos on "outliers" found in *dest = bs->getBitsNoFill(dsc.bps); (for the new v5 raw format) might be interesting?

from rawspeed.

AFAIK, there is no such thing as 'straight sensor data' today. Many (most? all?) cameras process raw values before recording (easily see on noise spectrum plot, or on just noise vs exposure time plot)

from rawspeed.

I mostly agree with #137 (comment).
There is no definition of what 'straight sensor data' is, so it can be argued either way.

Some of ISO push/pull may be done after readout and ADC, some cameras adjust noise in raws,
some cameras produce multiple exposures combined in various ways as raw,
a few of [super dumb] cameras even correct vignetting in raws ...

from rawspeed.

I have created https://www.dpreview.com/forums/post/61187570 to rattle a chain on zero data in high-resolution raws in the G9.

from rawspeed.

It might now be interesting to employ https://www.rdocumentation.org/packages/adimpro/versions/0.8.2 and then have a look at the individual channels, again, for outliers?

from rawspeed.

OK, that means that one should use panasonic-specified cropping, not 'maximum visible pixels area'.
Good point.
I'm too busy now with other tasks, I'll return to your samples (and panasonic cropping) in a few days.

from rawspeed.