get_wmt18_seg_result question about bert_score HOT 3 OPEN

wmt18_log.csv

from bert_score.

Hi, I've also got the same result as @Hou-jing using RoBERTa-large, and I'm also wondering why the numbers are different from the reported scores in the paper. I also tried to reproduce the results on WMT17-seg and got different (slightly better) results (0.7614 on tr-en), so I'm assuming the implementation has changed a bit. Would @felixgwu @Tiiiger or someone clarify this?

from bert_score.

Hi @Hou-jing and @twadada,

The script get_wmt18_seg_result.py is for WMT18 segment-level kendall correlation scores (Table 4 in the paper) and another script get_wmt17_sys_results.py is for WMT17 system-level pearson correlation scores (Tables 15 and 17 in the paper).
To get WMT18 system-level scores, you'll need to modify get_wmt17_sys_results.py to use WMT18 data.

Admittedly, we observe that these scripts give slightly different scores compared to the paper. We found that BERTScores can change slightly with newer pytorch and huggingface's transformers versions. We will look into this.

from bert_score.