keisks / jfleg Goto Github PK

Most times I run this script (though not every time), I get:
.../python2.7/site-packages/scipy/stats/_distn_infrastructure.py:1910: RuntimeWarning: invalid value encountered in multiply lower_bound = self.a * scale + loc
.../python2.7/site-packages/scipy/stats/_distn_infrastructure.py:1911: RuntimeWarning: invalid value encountered in multiply upper_bound = self.b * scale + loc
[['0.584945', '0.000000', '(nan,nan)']]
due to std passed to scipy.stats.norm.interval(0.95, loc=mean, scale=std) in get_gleu_stats being 0.0 (because the scores are either all the same, or a list of size 1).

1. I think you should add this paper to the leaderboard: https://arxiv.org/pdf/1807.01270.pdf
   It only reports JFLEG test GLEU, not dev, but it is beating the reference score.

2. Just so I understand better, I see that this repo includes dev and test data, but not train data. Where can I find train data? or, do I need to train on a different source and only evaluate on this data?

Thanks

A cased version of test.spellchecked.src would be helpful (like it is in dev), since many papers any way use a spellchecker.
Thanks

I tested the gleu.py script and got these 4 values. I know that the last 2 values (0.647,0.649) are confidence values. However, I'm not sure about the first 2 values which is (0.668878 and 0.010937). Upon checking the script, i know that one of them is the average but i don't know which is which. I've been reading the docs and ReadME files and I can't find any information on how to decipher the 4 output value. It would be better if you can add a label besides the values or update the docs regarding the 4 output values of GLEU.py script. Thank you