Unexpectedly bad performance on bart-base about rebel HOT 7 CLOSED

Glad I could help, and thanks for bringing it up :)

from rebel.

That doesn't make sense indeed. We did some experiments with bart-base and while performance was lower, it wasn't so dramatically lower.

I suspect this may be due to some of the differences between the configs for bart-base and bart-large.
Perhaps you are keeping the tokenized cached datasets with bart-large and training on those with bart-base?

from rebel.

I don't think that is the issue. I re-checked by setting force_download = True just to make sure. It's still the same.

from rebel.

I meant in the dataset. Try overwrite_cache=True in the hydra config or in the training command.

from rebel.

That one was set correctly as well.

from rebel.

Unfortunately that was my main guess and was wrong. There's also been some recent changes to the BART config, but I believe they only affected bart-large and not the base model (see huggingface/transformers#15559)

Perhaps it has something to do with this:
huggingface/transformers#9731

There must be some config/tokenizer issue since those performances are too far apart from each other.

from rebel.

@LittlePea13 Thank you so much for your help. The problem was indeed with huggingface/transformers#15559. I downloaded a previous version of bart-base, and the issue is fixed now:

processed 288 sentences with 421 relations; found: 379 relations; correct: 263.
	ALL	 TP: 263;	FP: 113;	FN: 143
		(m avg): precision: 69.95;	recall: 64.78;	f1: 67.26 (micro)
		(M avg): precision: 71.62;	recall: 66.88;	f1: 68.99 (Macro)

	killed by: 	TP: 41;	FP: 8;	FN: 6;	precision: 83.67;	recall: 87.23;	f1: 85.42;	49
	residence: 	TP: 62;	FP: 37;	FN: 36;	precision: 62.63;	recall: 63.27;	f1: 62.94;	99
	location: 	TP: 53;	FP: 17;	FN: 36;	precision: 75.71;	recall: 59.55;	f1: 66.67;	70
	headquarters location: 	TP: 60;	FP: 31;	FN: 36;	precision: 65.93;	recall: 62.50;	f1: 64.17;	91
	employer: 	TP: 47;	FP: 20;	FN: 29;	precision: 70.15;	recall: 61.84;	f1: 65.73;	67
Testing: 100% 18/18 [00:14<00:00,  1.27it/s]
--------------------------------------------------------------------------------
DATALOADER:0 TEST RESULTS
{'test_F1_micro': 67.26342710997442,
 'test_loss': 0.2752641439437866,
 'test_prec_micro': 69.94680851063829,
 'test_recall_micro': 64.77832512315271}
--------------------------------------------------------------------------------

I'm honestly not exactly sure why this would happen, but at least it works :)

from rebel.