Reproductibility issue about unilm HOT 5 CLOSED

Thanks for spotting the incorrect script path.

-li

from unilm.

Hi @colanim ,

Could you also post the evaluation command? It seems that the evaluation script is not correctly executed.

-li

from unilm.

I used :

python gigaword/eval.py --pred ${MODEL_RECOVER_PATH}.${EVAL_SPLIT}.alp1.0
  --gold ${DATA_DIR}/org_data/${EVAL_SPLIT}.summary --trunc_len 70 --perl

where

echo $MODEL_RECOVER_PATH

/root/code/unilm/src/cnndm_model/cnndm_model.bin

echo $EVAL_SPLIT

test

echo $DATA_DIR

/root/code/unilm/src/cnndm_data

from unilm.

It seems to be from my pyrouge installation.

When I run : (without --perl option)

python gigaword/eval.py --pred ${MODEL_RECOVER_PATH}.${EVAL_SPLIT}.alp1.0
  --gold ${DATA_DIR}/org_data/${EVAL_SPLIT}.summary --trunc_len 70

ROUGE-1: 0.4330151757396155 ROUGE-2: 0.1900106125427025

However I would like to make it work with the official ROUGE script...
I installed pyrouge differently than you did in your README (unit tests of pyrouge were not passing when I followed steps of the README). I followed this answer to install pyrouge.

from unilm.

8c1f46d fixed my problem : I was using wrong script.

Using cnndm/eval.py instead of gigaword/eval.py fixed it.

I could reproduce your result, using official ROUGE script.

Thanks a lot of the help !! 👍

from unilm.