Does it Work on FP16 with latest nvidia amp？ about spanbert HOT 6 OPEN

In FP16 , output is nan , and in mid layer(maybe 17th or 18th) in the bert , I found that the attention is -3000，and softmax result is nan

from spanbert.

When I tried to finetune spanbert large on my own task using fp16 with the "amp" module from the apex, I meet a similar error.

I am using the code from huggingface and tried both using "Spanbert/spanbert-large-cased" and the model binaries provided in this repo. Both gives the identical result, which kept telling me gradient overflow and rescaling the loss.

Interestingly, when I tried to replace spanbert-large with BERT base/large, and spanbert-base model, these models work perfectly and achieved expected results.

Also, the spanbert-large work very well when I turn off the fp16 training.

Here I found this guy cannot run fp16 with spanbert on another task.

In a conclusion, I guess that the spanbert-large model may work with the new Nvidia amp

from spanbert.

I will need more information to help you debug. Could you please include what command you're running?

from spanbert.

YES ， I use {https://dl.fbaipublicfiles.com/fairseq/models/spanbert_hf.tar.gz} to runing Huggingface`s run_squad , it Failed
Does it work with the latest amp？

from spanbert.

We haven't tested this with the new HF code. ICYMI, there's a run_squad.py in this repo. I'd recommend using that.

from spanbert.

Met same issue here, is there any solution yet?

from spanbert.