Improve your state of the art by using best activation function and best meta optimizer about gpt-3 HOT 5 OPEN

At the level these models are trained at, using a specific optimizer/activation will not necessarily get you better results.

from gpt-3.

Additionally considering GPT-3 size I would suggest not to use any optimizer above SGD because of the computation levels. Same goes for Mish.

from gpt-3.

@minimaxir it will not necessarily bring gains but it is still a low hanging fruit that should be tried.

from gpt-3.

@digantamisra98 RAdam (not the full Ranger package) does not increase computational cost.

I've read somewhere that Mish can be as efficient as Relu
Maybe with https://github.com/thomasbrandon/mish-cuda?

from gpt-3.

@LifeIsStrange everything above SGD is expensive.

from gpt-3.