openai / gpt-3 Goto Github PK

it would be appreciated if there is a chinese model available

Great work by the OpenAI team! The paper does not discuss it, so I'll be the first to ask:

What's the release plan for the model definition & weights? Will it be tiered by size, like GPT-2?

You could increase GPT 3 accuracy by using Ranger, which combine state of the art optimizers + gradient centralization
https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer
You seem to be using the Adam optimizer. It has been succeeded by RAdam (rectified Adam). Ranger will bring you this improvment and a lot more synergistic others, for free.

Hortogonally, you would probably benefit from Mish too instead of the one you use (Relu ?) but should be tested after Ranger as it could regress accuracy (even if unlikely)
https://github.com/digantamisra98/Mish