In our design for the project, part of the Augmentation block was to have a trainable categorical distribution of DA operators from which sample during each iteration for each given data point x.

However, sampling a value from a distribution is differentiable. The common practice is to use Gumbel-Softmax to approximate the distribution such that it is differentiable and trainable through the backdrop.

Useful sources:

• What type of functions can/cannot be handled by backprop?
• What is Gumbel-Softmax?
• Categorical Representation with Gumbel-Softmax
• Pytorch implementation of Gumbel-Softmax

The current implementation puts equal weights on each CE term, giving too many weights to augmented data when the ratio of original data to augmented data is 1:7.

The weights for these terms could be a tunable parameter; however, this could largely fall out of the scope of this project given the time constraints. One potential change we could make is to weight the CE losses as follows:

CE(M(x), y) + (1 / 7) * [CE(M(aug_x_1), y) + … + CE(M(aug_x_7), y)]

Summary

Rough draft of the experiment settings:

To compare the runtime, we will need to add timer to the code and run through Rotom's experiments as well.