• Focal loss https://arxiv.org/pdf/1708.02002.pdf

integrate frameworks to search for hyper params such as learning rate, warmups, etc

https://openreview.net/pdf?id=7Yhok3vJpU

Label smoothing https://paperswithcode.com/method/label-smoothing

• model.args.pretrained=True/False -- whether to initialize pretrained or not
• train.train_parent_after={type:int} -- train parent after this step
• train.min_steps={type:int} -- minimum steps to train -- ignore early stop until this step

1. try weighing at the mini-batch level

loss = 1/N \sum ...
is flawed when class imbalance.

Aggregate loss per class and take the macro average over class losses.

MLM task