segment-level doesn't seem to work on CLI (and also even if it did, would likely be misleading) about bert_score HOT 2 CLOSED

hi @hal3, thank you for your interest, big fan of your research here.

I think there are some misunderstandings. There are two separate issues going on here:

1. usage of -s
2. output order of bert_score

For 1)
if you run this command in the bert_score repo,

bert-score -r example/refs.txt -c example/hyps.txt --lang en -s

In the standard output I see

roberta-large_L17_no-idf_version=0.3.0(hug_trans=2.4.1) P: 0.957379 R: 0.961325 F1: 0.959333
0.984330        0.982384        0.983356
0.983224        0.973286        0.978230
0.912039        0.920429        0.916215
0.957694        0.960054        0.958873
0.960253        0.974826        0.967485
0.967353        0.968607        0.967979
0.952560        0.968049        0.960242
0.966578        0.965992        0.966285
0.939281        0.948455        0.943845
0.950472        0.951170        0.950821

The first line is the average score and following lines are segment level scores.
So I think the -s should be working.

For 2)

Let me explain the code and give you a minimal example later.

In bert_score, we dedup and sort when computing the embedding as you noticed in this line
https://github.com/Tiiiger/bert_score/blob/master/bert_score/utils.py#L355

however, after we cache the contextual embedding, we will reload it in the original order and then return. this can be seen in this line https://github.com/Tiiiger/bert_score/blob/master/bert_score/utils.py#L399

so here's a minimal example.

import bert_score
from bert_score import score

cands = [
    "how are you hal", "how are you hal", "have a"
]

refs = [
    "how are you hal", "how are you hal", "nice day"
]

P, R, F1 = score(cands, refs, lang='en')

print(F1)

Running the above code I get

tensor([1.0000, 1.0000, 0.8733])

As you see the output is not deduped (otherwise there will be 2 numbers) and sorted (otherwise the short sentence will come first).

Hope you find this reply helpful. If this is still bugging you, can you let me know your installed version and give me an minimal example so I can reproduce it.

from bert_score.

I am closing now but feel free to reopen if it's not working for you.

from bert_score.