Hey there,

I was thinking that a Pytorch/Pyro implementation of PCGrad may be interesting since an official implementation by the authors is nowhere in sight, what do you think? Seems like an interesting PyTorch abuse! 😉

PCGrad is a form of gradient surgery that projects a task’s gradient onto the normal plane of the gradient of any other task that has a conflicting gradient, which achieves substantial gains in efficiency and performance on a range of supervised multi-task learning and multi-task reinforcement learning domains. Moreover, it is model-agnostic and can be combined with previously-proposed multitask architectures for enhanced performance.

Repo with TF implementation: https://github.com/tianheyu927/PCGrad

Paper: https://arxiv.org/abs/2001.06782