Performance when fine-tuning on TACRED about spanbert HOT 5 CLOSED

The finetuned large models can be downloaded from here. Replace {task} with one of {squad1, squad2, tacred}
http://dl.fbaipublicfiles.com/fairseq/models/spanbert_{task}.tar.gz

I'll look into the other things you mentioned this week.

from spanbert.

@svjan5 I might not be 100% correct - From what I can tell, a 0.8 drop caused by removing --fp16 is not too surprising.

from spanbert.

Quick update:
I downloaded and evaluated the fine-tuned model you provided and it works perfectly!
I also noticed that, by copying the configuration provided in the README, I've used spanbert-base as the base model instead of spanbert-large. After retraining with the large model I get results very close to the paper scores.

Thanks again for providing the fine-tuned models.

from spanbert.

Hi,
I am unable to reproduce results on TACRED even with spanbert_tacred model.
Firstly, I ran the following command for training the model:
setgpu 1; python code/run_tacred.py --do_eval --data_dir ~/tacred/data/json/ --model spanbert-large-tacred --train_batch_size 32 --eval_batch_size 32 --learning_rate 2e-5 --num_train_epochs 10 --max_seq_length 128 --output_dir tacred_dir --do_train

Then, to get performance on test split. I ran:
setgpu 1; python code/run_tacred.py --do_eval --data_dir ~/tacred/data/json/ --model spanbert-large-tacred --train_batch_size 32 --eval_batch_size 32 --learning_rate 2e-5 --num_train_epochs 10 --max_seq_length 128 --output_dir tacred_dir --eval_test

The performance on test is as follows:

***** Eval results ***** (Test)
accuracy = 0.8775549680830486
eval_loss = 0.50735904131968
f1 = 0.699196667658435
precision = 0.691786870768325
recall = 0.706766917293233

The obtained F1 score is around 0.8 less than what is reported. The only thing I changed is removing --fp16 argument because the code gives an error with that as reported above. @ChristophAlt solution does not work in my case. Can removing --fp16 lead to a reduction in performance or am I missing something else.

Thanks in advance!

from spanbert.

Thanks @danqi! with --fp16 I got around 70.33 which is reasonably close. If possible please give some idea how including --fp16 is changing performance.

from spanbert.