Low scores when running scripts for BART about policyie HOT 3 OPEN

Yes, copying the file is ok. I am not sure why are you getting such a low score. It seems like the model is not trained appropriately. Although the code we released is verified, I can try to reproduce the paper results again and share the fine-tuned checkpoint. But it needs some time.

from policyie.

That would be really useful, thanks!
Would it be possible for you to reproduce the results on a Colab Notebook? This would enable me and everyone else to reproduce the training procedure as well :)

[One more difference I can think of is that the code wouldn't run installing fairseq==0.9.0 as specified, I had to install fairseq==0.10.2]

from policyie.

I also found this issue following identical steps in the training script for bart, (with fairseq==0.9.0), here are my scores

***** Eval results *****
{
    "all_slot_f1": 10.7,
    "all_slot_precision": 11.13,
    "all_slot_recall": 11.43,
    "intent_f1": 0.0,
    "type_II_exact_match": 0.0,
    "type_II_slot_f1": 27.9,
    "type_II_slot_precision": 28.13,
    "type_II_slot_recall": 27.81,
    "type_I_exact_match": 0.0,
    "type_I_slot_f1": 14.61,
    "type_I_slot_precision": 14.64,
    "type_I_slot_recall": 15.85
}

I wonder how it is going at the moment? Did you resolve the reproduction issue? Or could provide some pointers?

Thank you!

from policyie.