What is the necessity of using mse loss instead of cross entropy? about mixmatch HOT 14 CLOSED

1. We use the Brier score (MSE loss) to make the training more robust to incorrect guesses. This is detailed in the paper on arxiv.
2. You could use a big batch. Just be mindful that with training=True the batch size must always be the same for batch_norm to behave properly. In particular, we run the model in guess_labels (on u1, u2) and we run it in the main body again (on x', u1', u2'). That's why we ended up running each group (x', u1, u2, u1', u2') separately since combining them would result in two mismatched batches.

from mixmatch.

In the settings of this code base, there are 5 forward computations per iteration: 2 for label guessing on u1 and u2, and 3 on the mixed input. All of these 5 forward computations would affect the running mean/var status of bn layers, though computation on u1 and u2 are not meant for training. Would it be better if we find a way to remove the impact on bn status brought by u1 and u2?

from mixmatch.

Actually only one forward computation affects the batch norm running mean and var. We explicitly make sure of that in the code by shuffling the batches contents (layers.interleave) and by ignoring the global updates (see post_ops) other than for that one batch.

from mixmatch.

Thanks, I will try it.

from mixmatch.

Just be mindful that with training=True the batch size must always be the same for batch_norm to behave properly.

Hi, thank you for your in-depth discussion. With regard to the statement above, I wonder is there any literature shows that it is best to ensure that the batch sizes are consistent for infering through the model during one batch iteration?

from mixmatch.

I am not sure about literature on the topic. It's just that with different batches sizes, I would expect the variance to change. So basically I erred on the safe side in the code. But feel free to experiment may it can be simplified.

from mixmatch.

Thanks for reply.

from mixmatch.

Hi, I have one more question about the CIFAR100 experiment.
In your paper, a 28-layer Wide Resnet model (which has 135 filters per layer, resulting in 26 million parameters) is used for CIFAR100. According to the tutorial in readme file, I just set the parameter filter equals to 135 and the code shows that total number of parameter is 25,985,290. I also notice that the filter number of the first conv layer(filter=16) is fixed in your ResNet class . so I wonder if this setup is the same as yours for getting the results in your paper. Thanks.

from mixmatch.

Yes it is the same setup. In reality we had a little bit less parameters than the methods we compared to, so yes the number is correct it's almost 26M.

from mixmatch.

Thanks

from mixmatch.

Hi,
Thanks for these replies, I am not familiar with tensorflow, so I feel maybe I better ask a bit more about the details :)

Did you use a batch of 64 interleaved samples to train the model with the other 128 samples neglected in computing the gradients of all parameters (or only use 64 samples to update bn, and use all the 192 samples to update other parameters)? If only 64 samples are used on all parameters, since I want to try a big batch of 192 samples per forward computation, should I multiply the learning rate by 3? As for the number of epoches and iterations, you trained with batch size of 64 for 1024 epoches and with each epoch consists of 1024 iterations, so if I train with 192 samples per iteration, I should reduce the number of iterations per epoch to 1024/3, am I correct ?

from mixmatch.

All batches (192 samples) are used in gradient computation.

from mixmatch.

Hello, I am very inspired to read your thesis, but when I run the code, I find that the code is missing files. Can you send me the complete code? Appreciate it!

from mixmatch.

I'm closing this issue since I answered the original question. For unrelated questions, please open new issues.

from mixmatch.