Is moving mean and variance in batchnorm layer updated properly? about maml HOT 4 CLOSED

Duplicate of #9,

Sorry for opening new a new ticket. I will close it.

from maml.

Ok. And, yes, only beta is being used because the scaling can be handled by the scale of the weights in the following layer.

from maml.

Thanks for the such an instant response!

You mentioned at #9 that mean and variance of batch can be used instead of the tracked mean and variance, but, does Instance Normalization fit well in this situation? (Well, calculating statistics for each "batch" in the few-shot learning case does make sense in that "batch" is the whole dataset in this case...)

Any thoughts on this?

from maml.

You could certainly try instance norm. It would be pretty easy to add, and the statistics only rely on the current example. The implementation would look very similar to the layer norm implementation in the code.

from maml.