pixelBuffer normalization function? about coremlhelpers HOT 13 CLOSED

Why do you need to do this? It's better to let Core ML handle this for you.

from coremlhelpers.

New to this arena, so excuse my lack of understanding.

If my model is normalized via CoreML, input images must be similarly normalized? What is the best/easiest means to normalize a UIImage?

Brian

from coremlhelpers.

When you convert your model to Core ML you tell it how to normalize the image (by passing in the appropriate parameters). Then in your app you just use a regular CVPixelBuffer and Core ML takes care of the normalization. Even easier is using Vision, which also resizes the image if necessary.

from coremlhelpers.

Hi @hollance

Google just showed me this thread while searching image normalization and CoreML. Write now i'm in the process to convert a pytorch model through onnx. Original model normalizes each channel individually by subtracting average and dividing by standard deviation.
In this scenario it is not clear yet to me how to do it with coremltools through bias and scale factors. If it is not, @appsird makes makes sense.

from coremlhelpers.

The bias is for subtracting, and you can provide a separate bias for each of the 3 channels. The scale is for dividing by the standard deviation.

See also: http://machinethink.net/blog/help-core-ml-gives-wrong-output/

from coremlhelpers.

ok, but that model that i'm using has a standard deviation per channel.

IMAGE_NET_MEAN = [0.485, 0.456, 0.406]
IMAGE_NET_STD = [0.229, 0.224, 0.225]

This means that
normalizedValRed = (red - red_avg) / red_stddev
With bias you can do red - red_avg. Scale, though, i did not figure out yet how to apply it per channel.

from coremlhelpers.

Wait. This makes no sense.
What eventually makes sense is to replace input model and use MLMultiArray with normalized values. In the end probably the best is to keep input as image and add a custom layer to do the normalization.

from coremlhelpers.

Yeah it gets a bit trickier. But really I would just use 0.225 for all of them since they're so similar it probably doesn't matter.

from coremlhelpers.

That is a dangerous assumption...
I'm moving now to the snakes world and investigating how to add custom layers to the graph through coremltools. I really don't understand why Apple does not provide this type of normalization. All python frameworks have it and, together with (x-min)/(max - min), (x-avg)/stdev is one of the most common normalizations adopted in data mining methods. Doing it per color channel seems quite reasonable.

from coremlhelpers.

You're going to lose more precision due to 16-bit floating point precision issues than because of a difference of 0.001 or 0.004 in the standard deviation. So, I wouldn't lose any sleep over it.

You could probably achieve what you want by adding a ScaleLayerParams as the first layer in the model, although I'm not 100% sure that it accepts a different scale factor per channel.

from coremlhelpers.

@hollance and folks reading this.
There is an example now in coremltools how to scale per channel

https://github.com/apple/coremltools/blob/master/examples/Image_preprocessing_per_channel_scale.ipynb

cheers

from coremlhelpers.

Nice, thanks for finding this!

from coremlhelpers.

This was created a few days ago and added to master yesterday. Here you can find the flow

onnx/onnx-coreml#338

from coremlhelpers.