TACRED train F1 about spanbert HOT 6 CLOSED

Maybe this is because I used float32? Do I have to use float 16?

from spanbert.

That should not make a difference. Sorry about the delay. I'll look into it this week.

from spanbert.

@WindChimeRan

Besides replacing that line of code,

Can you post the command that you have run the code and how many epochs/updates did you run and get this F1 score?

I can take a look.

from spanbert.

Update:

If I put the training evaluation line into https://github.com/facebookresearch/SpanBERT/blob/master/code/run_tacred.py#L498, then it's ok. However, I can still not figure out why my first version failed.

python code/run_tacred.py \
  --do_train \
  --do_eval \
  --data_dir <TACRED_DATA_DIR> \
  --model spanbert-base-cased \
  --train_batch_size 32 \
  --eval_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 10 \
  --max_seq_length 128 \
  --output_dir tacred_dir \

I keep all the setting the same with the README.md except for fp16. Yes, fp 16 does not make difference.

The score of spanbert-large on training set is:

Precision (micro): 93.094%
Recall (micro): 96.034%
F1 (micro): 94.541%

from spanbert.

I am still confused. Did you pass --eval_test or not? If you are going to change line #498, you have to add --eval_test otherwise it would still use the dev examples.

Do you think this is still an issue or I can close it?

from spanbert.

#498 is the main change. I also did many minor changes that I cannot remember all of them.

Thanks for your reply.

from spanbert.