Evaluate metrics in WMT task about tensor2tensor HOT 2 CLOSED

We added utils/get_ende_bleu.sh script that has the commands we used to go from detokenized decodes (produced by t2t_trainer --decode_from_file) to BLEU. It requires MOSES and perl, so you might need to look into the script and adjust paths to run it. But it's probably the best answer to your questions:
(1) the script provided (which uses MOSES tokenizer and multi-bleu)
(2) It is case-sensitive (coming from MOSES)
(3) Yes, the script if for de-tokenized output produced by the trainer on XXX_tokens_32k

• If you're running on wmt_ende_bpe32k then instead of the tokenizer call in the script, do this:
  perl -ple 's{@@ }{}g' > $decodes_file.target
  (4) This is hard, because it needs perl and MOSES and we don't want to call them during training
  (it's esp. a problem in the distributed setting, where machines don't have MOSES might not have perl).
  That's why we have our approximate BLEU metric that gives us an idea where we are.

Hope that helps, feel free to reopen with more questions!

from tensor2tensor.

4. Is it possible to run an evaluation script during training after every n_th step?

from tensor2tensor.