Comments (3)
Fixed. I can directly set the path in the first line of the install ROUGE-1.5.5.pl.
BTW, I found a similar problem like #43 . I follow the Readme of BART for reproducing CNN/DM results.
I directly use the public checkpoint for CNN/DM, so maybe there is no problem in training/inference.
I use the Corenlp to tokenized the reference and prediction file.
I use the pyrouge or files2rouge to evaluate.
The pyrouge results are below:
`1 ROUGE-1 Average_R: 0.51297 (95%-conf.int. 0.51018 - 0.51576)
1 ROUGE-1 Average_P: 0.40694 (95%-conf.int. 0.40444 - 0.40952)
1 ROUGE-1 Average_F: 0.44288 (95%-conf.int. 0.44063 - 0.44507)
1 ROUGE-2 Average_R: 0.24449 (95%-conf.int. 0.24151 - 0.24725)
1 ROUGE-2 Average_P: 0.19458 (95%-conf.int. 0.19220 - 0.19700)
1 ROUGE-2 Average_F: 0.21127 (95%-conf.int. 0.20876 - 0.21368)
1 ROUGE-L Average_R: 0.35718 (95%-conf.int. 0.35430 - 0.35986)
1 ROUGE-L Average_P: 0.28160 (95%-conf.int. 0.27928 - 0.28400)
1 ROUGE-L Average_F: 0.30727 (95%-conf.int. 0.30490 - 0.30957)`
This R-L result is very lower the expected 40.90.
The files2rouge results are below:
1 ROUGE-1 Average_R: 0.40390 (95%-conf.int. 0.40134 - 0.40648)
1 ROUGE-1 Average_P: 0.50956 (95%-conf.int. 0.50689 - 0.51227)
1 ROUGE-1 Average_F: 0.43971 (95%-conf.int. 0.43750 - 0.44215)
1 ROUGE-2 Average_R: 0.19393 (95%-conf.int. 0.19147 - 0.19646)
1 ROUGE-2 Average_P: 0.24389 (95%-conf.int. 0.24102 - 0.24680)
1 ROUGE-2 Average_F: 0.21063 (95%-conf.int. 0.20813 - 0.21326)
1 ROUGE-L Average_R: 0.37440 (95%-conf.int. 0.37190 - 0.37693)
1 ROUGE-L Average_P: 0.47246 (95%-conf.int. 0.46979 - 0.47529)
1 ROUGE-L Average_F: 0.40764 (95%-conf.int. 0.40535 - 0.41010)
We can observe the difference on R-L. Could you give some advice for this situation?
from files2rouge.
I don't know how pyrouge works but if I remember right we calculate sentence level ROUGE-L (check original article for reference). Maybe pyrouge does not split sentence correctly, thus giving different results.
from files2rouge.
Thanks a lot.
from files2rouge.
Related Issues (20)
- A stock when I run the final code
- Problem with stemming HOT 1
- Tiny Suggestion: change tempfile.mkdtemp() to TemporaryDirectory() HOT 1
- Inconsistent Behavior after Recent Update: Default Value of Stemming Option HOT 1
- Incorrect Order of Reference and Summary in Readme HOT 9
- Problem with Average_F HOT 1
- Incorrect L-scores HOT 1
- TypeError: __init__() got an unexpected keyword argument 'stemming' HOT 1
- PermissionError: [Errno 13] Permission denied: '/home/Bwwhst28Y/anaconda3/envs/pytorch/lib/python3.6/site-packages/files2rouge-2.1.0-py3.6.egg/files2rouge/ROUGE-1.5.5.pl' HOT 4
- Cannot open exception db file for reading HOT 1
- TypeError: __init__() got an unexpected keyword argument 'log_level' HOT 2
- fscore does not match with recall and precision HOT 1
- files2rouge tests fail due to not finding installed XML::Parser module
- rouge-su HOT 1
- ValueError: Model and System line counts must match, 196 != 1512 HOT 1
- Cannot open exception db file for reading, even with read permission HOT 1
- "eos" use? HOT 2
- TypeError: __init__() got an unexpected keyword argument 'log_level' HOT 1
- Is this repo got exactly same result with google/seq2seq official repo?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from files2rouge.