Comments (3)
from bert_score.
Hi, I've also got the same result as @Hou-jing using RoBERTa-large, and I'm also wondering why the numbers are different from the reported scores in the paper. I also tried to reproduce the results on WMT17-seg and got different (slightly better) results (0.7614 on tr-en), so I'm assuming the implementation has changed a bit. Would @felixgwu @Tiiiger or someone clarify this?
from bert_score.
The script get_wmt18_seg_result.py
is for WMT18 segment-level kendall correlation scores (Table 4 in the paper) and another script get_wmt17_sys_results.py
is for WMT17 system-level pearson correlation scores (Tables 15 and 17 in the paper).
To get WMT18 system-level scores, you'll need to modify get_wmt17_sys_results.py
to use WMT18 data.
Admittedly, we observe that these scripts give slightly different scores compared to the paper. We found that BERTScores can change slightly with newer pytorch and huggingface's transformers versions. We will look into this.
from bert_score.
Related Issues (20)
- Warning HOT 1
- BERTScore with covid-twitter-bert
- Padding token ID HOT 2
- Run bert_score on TPU (google colab) instead of gpu.
- Calculate embeddings once HOT 1
- Tensor size error with multiple refs tweets
- Can bertscore calculate semantic similarity between languages?
- AttributeError: 'RobertaTokenizerFast' object has no attribute 'max_len' HOT 1
- Different hug_trans version cause different BertScore HOT 6
- BertScore giving different results each time
- Semantic similarity between essays and a theme HOT 1
- Model and Tokenizer change HOT 2
- not able to build wheel
- Question regarding semantic correctness of sentences
- Trying Custom BERT model HOT 1
- Coco challenge link expired
- Why is matplotlib needed?
- Nan of P score?
- Tensor dimension mismatch when using lang='en-sci' HOT 2
- Cache embeddings HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bert_score.