FP16 for training acceleration about lxmert HOT 2 CLOSED

Thanks for your attention! I will explore this possibility in the next few days and let you know. I will release it with either --fp16 option or a new branch.

from lxmert.

Hi,

The half-precision support with apex library is now available for pre-viewing. You could switch to it by

git checkout fp16

, and adding argument --fp16 to any bash commands would run with apex, e.g.,

bash run/vqa_finetune.bash 0 vqa_lxr955_tiny --tiny --fp16

I would really appreciate it if you could help test it!

Comments:

• Fine-tuning results with fp16 has been verified: VQA: 70.12, GQA: 59.81, NLVR2: 74.26. Pre-training with fp16 has not been verified.

• Performance: VQA/GQA/Pre-training's speed is not changed, but the memory is reduced. NLVR2 gets 2x speed gain.

from lxmert.