Batch normalization about maml HOT 8 CLOSED

@cbfinn, as you mentioned before,

I compute the test-time statistics using the test batch of data, instead of computing the average training statistics

This seems to be a bit of cheating especially on test-time. In general, we can assume evaluating only one sample at a time on test-time and then there is no way to get proper statistics for batch_norm. This means the test-set performance will partially dependent on the size of batch.

from maml.

I compute the test-time statistics using the test batch of data, instead of computing the average training statistics. This doesn't require keeping track of batch norm training statistics. [Note that train is always set to True when calling the batch_norm function, which means that tensorflow will compute the statistics using the current batch]

It's possible that it would work better by using training batch statistics, but I haven't tried it.

from maml.

It's not the issue I'm talking about. See the Note from tf.contrib.layers.batch_norm page

Note: when training, the moving_mean and moving_variance need to be updated. By default the update ops are placed in tf.GraphKeys.UPDATE_OPS, so they need to be added as a dependency to the train_op.

In other words, without these steps I wrote in the issue, moving_mean and moving_variance doesn't update at all (even during training). Again, maybe I'm missing some other way you're updating them.

from maml.

You only need to update moving_mean and moving_variance if you use them. In this case, the batch norm statistics are being computed using the batch data rather than a moving average of the statistics (so they don't need to be updated).

from maml.

OK. Indeed, they're needed only during testing.
Thanks.

from maml.

from maml.

Hi, I see your approcach.
If I use moving average of the statistics by adding update_op into train ops, Then Need I set train=FALSE when testing use batch_norm function?

from maml.

from maml.