pltrdy / files2rouge Goto Github PK

View Code? Open in Web Editor NEW

191.0 4.0 53.0 211 KB

Calculating ROUGE score between two files (line-by-line)

License: MIT License

Python 4.38% Perl 95.47% Dockerfile 0.15%

rouge-metric rouge summarization nlp natural-language-processing

files2rouge's People

Contributors

Stargazers

Watchers

files2rouge's Issues

Problem while running setup_rouge.py

cloud-miracl@Cloud-Miracl:~/Bureau/Nouha/Projects/files2rouge$ python setup_rouge.py
files2rouge uses scripts and tools that will not be stored with the python package
where do you want to save it? [default: /home/cloud-miracl/.files2rouge/]/home/cloud-miracl/usr/local/anaconda/lib/python3.6/site-packages/files2rouge-2.0.0-py3.6.egg/files2rouge
Copying './files2rouge/RELEASE-1.5.5/' to '/home/cloud-miracl/usr/local/anaconda/lib/python3.6/site-packages/files2rouge-2.0.0-py3.6.egg/files2rouge'
Traceback (most recent call last):
File "setup_rouge.py", line 40, in
data = copy_rouge()
File "setup_rouge.py", line 33, in copy_rouge
shutil.copytree(src_rouge_root, path)
File "/home/cloud-miracl/usr/local/anaconda/lib/python3.6/shutil.py", line 315, in copytree
os.makedirs(dst)
File "/home/cloud-miracl/usr/local/anaconda/lib/python3.6/os.py", line 220, in makedirs
mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/home/cloud-miracl/usr/local/anaconda/lib/python3.6/site-packages/files2rouge-2.0.0-py3.6.egg/files2rouge'

Unexpected keyword argument 'log_level'

The log_level argument passed to pyrouge.Rouge155 is denied as unexpected argument:

Preparing documents...
Running ROUGE...
Traceback (most recent call last):
  File "/usr/local/bin/files2rouge", line 11, in <module>
    load_entry_point('files2rouge==2.0.0', 'console_scripts', 'files2rouge')()
  File "/usr/local/lib/python3.6/site-packages/files2rouge-2.0.0-py3.6.egg/files2rouge/files2rouge.py", line 65, in main
    log_level=log_level)
TypeError: __init__() got an unexpected keyword argument 'log_level'

Python: Version 3.6

when i run files2rouge summaries.txt references.txt i get this. My summaries.txt and references.txt have line by line entries

Preparing documents...
Running ROUGE...
Cannot open exception db file for reading: yes/data/WordNet-2.0.exc.db
Traceback (most recent call last):
  File "/Users/vishrutkohli/Desktop/keras-text-summarization/demo/files2rouge/env3/bin/files2rouge", line 11, in <module>
    load_entry_point('files2rouge==2.0.0', 'console_scripts', 'files2rouge')()
  File "/Users/vishrutkohli/Desktop/keras-text-summarization/demo/files2rouge/env3/lib/python3.6/site-packages/files2rouge-2.0.0-py3.6.egg/files2rouge/files2rouge.py", line 82, in main
    output = r.convert_and_evaluate(rouge_args=rouge_args_str)
  File "/Users/vishrutkohli/Desktop/keras-text-summarization/demo/files2rouge/env3/lib/python3.6/site-packages/pyrouge/Rouge155.py", line 367, in convert_and_evaluate
    rouge_output = self.evaluate(system_id, rouge_args)
  File "/Users/vishrutkohli/Desktop/keras-text-summarization/demo/files2rouge/env3/lib/python3.6/site-packages/pyrouge/Rouge155.py", line 342, in evaluate
    rouge_output = check_output(command, env=env).decode("UTF-8")
  File "/Users/vishrutkohli/anaconda3/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/Users/vishrutkohli/anaconda3/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['yes/ROUGE-1.5.5.pl', '-e', 'yes/data', '-c', '95', '-r', '1000', '-n', '2', '-a', '-m', '/var/folders/3h/7x3f6sqd1276lk8pz98k5k300000gn/T/tmpsvwqq4ob/rouge_conf.xml']' returned non-zero exit status 79.

"eos" use?

files2rouge/files2rouge/utils.py

Line 36 in d3d09c8

split_sen = (" %s\n" % eos).join(line.split(" %s" % eos))

Why add a space before eos?

A stock when I run the final code

Hi,
Here is the running result that I have got based on the "text-summarization-tensorflow" project.

1 ROUGE-1 Average_R: 0.31513 (95%-conf.int. 0.31392 - 0.31628)
1 ROUGE-1 Average_P: 0.41923 (95%-conf.int. 0.41779 - 0.42064)
1 ROUGE-1 Average_F: 0.34849 (95%-conf.int. 0.34722 - 0.34971)

1 ROUGE-2 Average_R: 0.14793 (95%-conf.int. 0.14693 - 0.14890)
1 ROUGE-2 Average_P: 0.19880 (95%-conf.int. 0.19764 - 0.19998)
1 ROUGE-2 Average_F: 0.16334 (95%-conf.int. 0.16228 - 0.16431)

1 ROUGE-L Average_R: 0.29737 (95%-conf.int. 0.29616 - 0.29849)
1 ROUGE-L Average_P: 0.39480 (95%-conf.int. 0.39344 - 0.39618)
1 ROUGE-L Average_F: 0.32844 (95%-conf.int. 0.32725 - 0.32961)

Elapsed time: 19583.610 seconds

But I spend more than 4 hours to run the evaluation part which is the command "file2rouge summaries.txt reference.txt" on Ubuntu 18.04.3 LTS without GPU. Does anyone knows why it takes so long? And how many times does you run in the task? Thanks.

SyntaxError: invalid syntax for print(file=saveto, *args, **kwargs)

i've got the error as the traceback below, when I try to run the command python setup_rouge.py

Traceback (most recent call last):
  File "setup_rouge.py", line 7, in <module>
    from files2rouge import settings
  File "/home/cerdas/files2rouge/files2rouge/__init__.py", line 2, in <module>
    from files2rouge.files2rouge import main
  File "/home/cerdas/files2rouge/files2rouge/files2rouge.py", line 17, in <module>
    from files2rouge import utils
  File "/home/cerdas/files2rouge/files2rouge/utils.py", line 16
    print(file=saveto, *args, **kwargs)
              ^
SyntaxError: invalid syntax

I have no idea whats wrong with the code, thanks.

files2rouge.py: error: unrecognized arguments: --summary --reference

excuse me, I install this fowling your step, but when I use it always go wrong, I don't kwon how to run it?

The difference Rouge-L between the pyrouge package and files2rouge

Hi, @pltrdy Thx for this nice rouge package～
But I run files2rouge got an error like:
Can't locate LWP/UserAgent.pm in @inc (@inc contains: /users/XXX/.files2rouge /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .)

/users/XXX/.files2rouge is my files2rouge setup path during the installation.
In fact, I install LWP/UserAgent.pm in my perl5 path, but I am not a root user so I can't install any perl package in /usr path.
Therefore, I want to know whether there is a way to set the perl path for rouge-1.5.5.pl?

Inconsistent Behavior after Recent Update: Default Value of Stemming Option

files2rouge was using stemming by default before the recent commit e39d882 because pyrouge sets stemming as True by default. However the stemming option is default by False now, which causes the ROUGE scores obtained by standard command files2rouge file1 file2 is not comparable to those obtained by pyrouge or the previous files2rouge, sometimes there is a large gap (over 1 point on R1 in my case).

This causes non-trivial trouble for us to reproduce other's results because many papers use either pyrouge or the previous files2rouge to compute ROUGE scores. I appreciate the update to make stemming controllable, but I think the behavior of files2rouge should be ideally consistent with before, or at least mention this change in the README, otherwise it is difficult for us to realize that the current files2rouge command is actually different from the file2rouge used by other papers published before the recent update (by default).

TypeError: init() got an unexpected keyword argument 'log_level'

tengweideMacBook-Pro:files2rouge tengwei$ files2rouge result.txt valid.title.filter.txt
Preparing documents...
Running ROUGE...
Traceback (most recent call last):
File "/anaconda/bin/files2rouge", line 11, in
load_entry_point('files2rouge==2.0.0', 'console_scripts', 'files2rouge')()
File "/anaconda/lib/python3.5/site-packages/files2rouge-2.0.0-py3.5.egg/files2rouge/files2rouge.py", line 65, in main
log_level=log_level)
TypeError: init() got an unexpected keyword argument 'log_level'

excuse me, I was successful running the files2rouge before. But this time I meet some issues. I don't know how to solve this problem, can you help me? Look forward to your reply. thanx!

ValueError: Model and System line counts must match, 196 != 1512

My hyps.txt and refs.txt each has 200 sentences, and each sentence contains lots of dot '.' and ends up by '\n'.

When I ran files2rouge hyps.txt refs.txt, it gives me ValueError: Model and System line counts must match, 196 != 1512

so I tried to change EOS sign from dot '.' to '\n' by enter files2rouge hyps.txt refs.txt -e '\n' or files2rouge hyps.txt refs.txt --eos '\n', both commands didn't work and gave me the same error ValueError: Model and System line counts must match, 196 != 1512.

I wonder if I entered the wrong commands or if it's because other issues in my text files? Any help would be greatly appreciated!

rouge-su

Hi,
I want to use rouge-su metric to evaluate my results , can this script compute rouge-su?

Thanks

anaconda python err

Can't load ROUGE settings in '/home1/txr/anaconda3/envs/py36/lib/python3.6/site-packages/files2rouge-2.0.0-py3.6.egg/files2rouge/settings.json'. Check that the file exists or initialize it with 'setup_rouge.py'

Cannot open exception db file for reading, even with read permission

Hi, I have encountered the same issue as #48 but I failed to fix it by giving the db file permission:

l /Users/huyiwen/.files2rouge/data/WordNet-2.0.exc.db
.rwxrwxrwx 12k huyiwen  6 Jul 21:43 /Users/huyiwen/.files2rouge/data/WordNet-2.0.exc.db

I'm working on macOS 12.4 and Python 3.8.12. I installed the module under the instruction of README. The error occurs both in command line and python script:

Preparing documents... 0 line(s) ignored
Running ROUGE...
Cannot open exception db file for reading: /Users/huyiwen/.files2rouge/data/WordNet-2.0.exc.db
Traceback (most recent call last):
  File "/usr/local/Caskroom/miniconda/base/envs/TextBox/bin/files2rouge", line 33, in <module>
    sys.exit(load_entry_point('files2rouge==2.1.0', 'console_scripts', 'files2rouge')())
  File "/usr/local/Caskroom/miniconda/base/envs/TextBox/lib/python3.8/site-packages/files2rouge-2.1.0-py3.8.egg/files2rouge/files2rouge.py", line 131, in main
    run(args.summary,
  File "/usr/local/Caskroom/miniconda/base/envs/TextBox/lib/python3.8/site-packages/files2rouge-2.1.0-py3.8.egg/files2rouge/files2rouge.py", line 92, in run
    output = r.convert_and_evaluate(rouge_args=rouge_args_str)
  File "/usr/local/Caskroom/miniconda/base/envs/TextBox/lib/python3.8/site-packages/pyrouge/Rouge155.py", line 368, in convert_and_evaluate
    rouge_output = self.evaluate(system_id, rouge_args)
  File "/usr/local/Caskroom/miniconda/base/envs/TextBox/lib/python3.8/site-packages/pyrouge/Rouge155.py", line 343, in evaluate
    rouge_output = check_output(command, env=env).decode("UTF-8")
  File "/usr/local/Caskroom/miniconda/base/envs/TextBox/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/local/Caskroom/miniconda/base/envs/TextBox/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/Users/huyiwen/.files2rouge/ROUGE-1.5.5.pl', '-e', '/Users/huyiwen/.files2rouge/data', '-c', '95', '-r', '1000', '-n', '2', '-a', '-m', '/var/folders/j8/sbwvtz_j2_s7mlsj4z3zbb7r0000gn/T/tmptncmkl1e/rouge_conf.xml']' returned non-zero exit status 79.

Thanks a lot for your help!

TypeError: init() got an unexpected keyword argument 'log_level'

Hi, thanks for the nice work.
when i use the files2rouge, i met this issue:

(testtransformers) [zggao@cu05 result]$ files2rouge test.hypo.tokenized test.target.tokenized
Preparing documents... 0 line(s) ignored
Running ROUGE...
Traceback (most recent call last):
File "/home/zggao/anaconda3/envs/testtransformers/bin/files2rouge", line 33, in
sys.exit(load_entry_point('files2rouge==2.1.0', 'console_scripts', 'files2rouge')())
File "/home/zggao/anaconda3/envs/testtransformers/lib/python3.6/site-packages/files2rouge-2.1.0-py3.6.egg/files2rouge/files2rouge.py", line 105, in main
args.stemming)
File "/home/zggao/anaconda3/envs/testtransformers/lib/python3.6/site-packages/files2rouge-2.1.0-py3.6.egg/files2rouge/files2rouge.py", line 54, in run
stemming=stemming)
TypeError: init() got an unexpected keyword argument 'log_level'

How to test rouge score of each sentence ?

For example, there is a text file containing about 1000 sentences. I want to test rouge score of per sentence not rather all sentences. If I put every sentence into a file, that is too hard.
Can I read one sentence at a time and then test rouge score ?

How to use files2rouge on Chinese words

Hello.:
When I use LCSTS data ,I also use file2rouge script to evaluate. However, I find the score is low. I would like to ask how to use file2rouge script to measure Chinese data.

I hope for your reply.

Problem with stemming

Hi,
I am trying to perform the stemming before computing the statistics. I added -m option in the args however I'm getting the same scores as without it.

Do not work with non english word (Russian)

This is the library does not work with Russian words.
I tried with the same reference and candidate texts:

я иду на работу

And getting this logs:

root@00046ff67f96:/etc/rouge# ./run  ./data/summary ./data/reference  -a "-c 96 -v  -n 2 -a"
Preparing documents... 0 line(s) ignored
Running ROUGE...
@Eval (1)
***P /tmp/tmpdgp6q634/system/s.0.txt

_cn_|0

_cn_|0
***M /tmp/tmpdgp6q634/model/m.A.0.txt
total 1-gram model count: 0
total 1-gram peer count: 0
total 1-gram hit: 0
total ROUGE-1-R: 0.00000
total ROUGE-1-P: 0.00000
total ROUGE-1-F: 0.00000
1.1
---------------------------------------------
1 ROUGE-1 Average_R: 0.00000 (96%-conf.int. 0.00000 - 0.00000)
1 ROUGE-1 Average_P: 0.00000 (96%-conf.int. 0.00000 - 0.00000)
1 ROUGE-1 Average_F: 0.00000 (96%-conf.int. 0.00000 - 0.00000)
@Eval (1)
***P /tmp/tmpdgp6q634/system/s.0.txt

_cn_|0

_cn_|0
***M /tmp/tmpdgp6q634/model/m.A.0.txt
total 2-gram model count: 0
total 2-gram peer count: 0
total 2-gram hit: 0
total ROUGE-2-R: 0.00000
total ROUGE-2-P: 0.00000
total ROUGE-2-F: 0.00000
1.1
---------------------------------------------
1 ROUGE-2 Average_R: 0.00000 (96%-conf.int. 0.00000 - 0.00000)
1 ROUGE-2 Average_P: 0.00000 (96%-conf.int. 0.00000 - 0.00000)
1 ROUGE-2 Average_F: 0.00000 (96%-conf.int. 0.00000 - 0.00000)
***P /tmp/tmpdgp6q634/system/s.0.txt

0:
LCS: -
***M /tmp/tmpdgp6q634/model/m.A.0.txt

0:
total ROUGE-L model count: 0
total ROUGE-L peer count: 0
total ROUGE-L hit: 0
total ROUGE-L-R score: 0.00000
total ROUGE-L-P: 0.00000
total ROUGE-L-F: 0.00000
1.1
---------------------------------------------
1 ROUGE-L Average_R: 0.00000 (96%-conf.int. 0.00000 - 0.00000)
1 ROUGE-L Average_P: 0.00000 (96%-conf.int. 0.00000 - 0.00000)
1 ROUGE-L Average_F: 0.00000 (96%-conf.int. 0.00000 - 0.00000)

Is this know's issue or I made some wrong?

Files with Different Number of Lines

Hi Pltrdy!

Do you know of any python ROUGE calculator that supports it if files have different number of lines?

Thanks!
Shirley

setup_rouge.py does not work with Python 3

On python 3.5:
python setup_rouge.py
files2rouge uses scripts and tools that will not be stored with the python package
Traceback (most recent call last):
File "setup_rouge.py", line 37, in
data = copy_ROUGE()
File "setup_rouge.py", line 20, in copy_ROUGE
path = raw_input("where do you want to save it? [default: %s]" % default_root)
NameError: name 'raw_input' is not defined

TypeError: init() got an unexpected keyword argument 'log_level'

hello there,
I have encountered the Error : TypeError: init() got an unexpected keyword argument 'log_level' , while trying to analyse some NanoPore seq data.
i have implemented the steps presented in this thread , #7 .
i have pyrouge 0.1.3 and ROUGE-1.5.5 is configured properly.

i am using this syntax in python 3 :
db = SampCompDB ( db_fn = "/media/shiberlab/My Passport/Analysis-MM/nanopolish/Analysis-MM/nanopolish/18.10.2022/outSampComp.db",fasta_fn = "/media/shiberlab/My Passport/Analysis-MM/nanopolish/GCF_000146045.2_R64_rna.fa",log_level="warning")

any ideas ?

TypeError: init() got an unexpected keyword argument 'stemming'

Hi,
On running the command: files2rouge reference.txt summary.txt

I get the following error:
Preparing documents... 0 line(s) ignored Running ROUGE... Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.8/bin/files2rouge", line 8, in <module> sys.exit(main()) File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/files2rouge/files2rouge.py", line 107, in main run(args.summary, File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/files2rouge/files2rouge.py", line 53, in run r = pyrouge.Rouge155(rouge_dir=os.path.dirname(s.data['ROUGE_path']), TypeError: __init__() got an unexpected keyword argument 'stemming'

P.S.:

I have installed Python3.8.7 and pip21.0.1
I get the same error even after adding the -nm flag to the above command.

OSError: [WinError 193] %1 is not a valid Win32 application ( i used python 3.6.5 32bit)

PS D:\LuanVan\Rouge\files2rouge> files2rouge summaries.txt references.txt
Preparing documents...
Running ROUGE...
Traceback (most recent call last):
  File "N:\python_86\Scripts\files2rouge-script.py", line 11, in <module>
    load_entry_point('files2rouge==2.0.0', 'console_scripts', 'files2rouge')()
  File "N:\python_86\lib\site-packages\files2rouge-2.0.0-py3.6.egg\files2rouge\files2rouge.py", line 82, in main
    output = r.convert_and_evaluate(rouge_args=rouge_args_str)
  File "N:\python_86\lib\site-packages\pyrouge\Rouge155.py", line 367, in convert_and_evaluate
    rouge_output = self.evaluate(system_id, rouge_args)
  File "N:\python_86\lib\site-packages\pyrouge\Rouge155.py", line 342, in evaluate
    rouge_output = check_output(command, env=env).decode("UTF-8")
  File "N:\python_86\lib\subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "N:\python_86\lib\subprocess.py", line 403, in run
    with Popen(*popenargs, **kwargs) as process:
  File "N:\python_86\lib\subprocess.py", line 709, in __init__
    restore_signals, start_new_session)
  File "N:\python_86\lib\subprocess.py", line 997, in _execute_child
    startupinfo)
OSError: [WinError 193] %1 is not a valid Win32 application

Following the instructions, got errors

This is what I got after python3 files2rouge.py ../summary.txt ../ref.txt --verbose:

Counting lines...
Refs: 810 lines	Summaries: 6 lines
Process Process-7:
Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "files2rouge.py", line 137, in _producer
    q.put([line, ref, summ, get_rouge(ref, summ, score=self.score)])
  File "files2rouge.py", line 35, in get_rouge
    ROUGE_path=ROUGE_path, data_path=ROUGE_data_path)
  File "/Users/.../files2rouge/pythonrouge/pythonrouge.py", line 65, in pythonrouge
    output = subprocess.check_output([ROUGE_path, "-e", data_path, "-a", "-m", "-2", "4","-n", "3", abs_xml_path], stderr=subprocess.STDOUT)
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/Users/.../files2rouge/pythonrouge/RELEASE-1.5.5/ROUGE-1.5.5.pl', '-e', '/Users/.../files2rouge/pythonrouge/RELEASE-1.5.5/data', '-a', '-m', '-2', '4', '-n', '3', '/var/folders/v4/vzt0mckn5zxgh70vbkx13pb80000gn/T/tmpeyd0wjw8/rouge.xml']' returned non-zero exit status 79.
Process Process-4:
Process Process-5:
Process Process-3:
Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "files2rouge.py", line 137, in _producer
    q.put([line, ref, summ, get_rouge(ref, summ, score=self.score)])
  File "files2rouge.py", line 35, in get_rouge
    ROUGE_path=ROUGE_path, data_path=ROUGE_data_path)
  File "/Users/.../files2rouge/pythonrouge/pythonrouge.py", line 65, in pythonrouge
    output = subprocess.check_output([ROUGE_path, "-e", data_path, "-a", "-m", "-2", "4","-n", "3", abs_xml_path], stderr=subprocess.STDOUT)
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/Users/.../files2rouge/pythonrouge/RELEASE-1.5.5/ROUGE-1.5.5.pl', '-e', '/Users/.../files2rouge/pythonrouge/RELEASE-1.5.5/data', '-a', '-m', '-2', '4', '-n', '3', '/var/folders/v4/vzt0mckn5zxgh70vbkx13pb80000gn/T/tmp1gidy0d3/rouge.xml']' returned non-zero exit status 79.
Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "files2rouge.py", line 137, in _producer
    q.put([line, ref, summ, get_rouge(ref, summ, score=self.score)])
  File "files2rouge.py", line 35, in get_rouge
    ROUGE_path=ROUGE_path, data_path=ROUGE_data_path)
  File "/Users/.../files2rouge/pythonrouge/pythonrouge.py", line 65, in pythonrouge
    output = subprocess.check_output([ROUGE_path, "-e", data_path, "-a", "-m", "-2", "4","-n", "3", abs_xml_path], stderr=subprocess.STDOUT)
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/Users/.../files2rouge/pythonrouge/RELEASE-1.5.5/ROUGE-1.5.5.pl', '-e', '/Users/.../files2rouge/pythonrouge/RELEASE-1.5.5/data', '-a', '-m', '-2', '4', '-n', '3', '/var/folders/v4/vzt0mckn5zxgh70vbkx13pb80000gn/T/tmpax5omjok/rouge.xml']' returned non-zero exit status 79.
Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "files2rouge.py", line 137, in _producer
    q.put([line, ref, summ, get_rouge(ref, summ, score=self.score)])
  File "files2rouge.py", line 35, in get_rouge
    ROUGE_path=ROUGE_path, data_path=ROUGE_data_path)
  File "/Users/.../files2rouge/pythonrouge/pythonrouge.py", line 65, in pythonrouge
    output = subprocess.check_output([ROUGE_path, "-e", data_path, "-a", "-m", "-2", "4","-n", "3", abs_xml_path], stderr=subprocess.STDOUT)
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/Users/.../files2rouge/pythonrouge/RELEASE-1.5.5/ROUGE-1.5.5.pl', '-e', '/Users/.../files2rouge/pythonrouge/RELEASE-1.5.5/data', '-a', '-m', '-2', '4', '-n', '3', '/var/folders/v4/vzt0mckn5zxgh70vbkx13pb80000gn/T/tmpaappf43x/rouge.xml']' returned non-zero exit status 79.
Process Process-6:
Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "files2rouge.py", line 137, in _producer
    q.put([line, ref, summ, get_rouge(ref, summ, score=self.score)])
  File "files2rouge.py", line 35, in get_rouge
    ROUGE_path=ROUGE_path, data_path=ROUGE_data_path)
  File "/Users/.../files2rouge/pythonrouge/pythonrouge.py", line 65, in pythonrouge
    output = subprocess.check_output([ROUGE_path, "-e", data_path, "-a", "-m", "-2", "4","-n", "3", abs_xml_path], stderr=subprocess.STDOUT)
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/Users/.../files2rouge/pythonrouge/RELEASE-1.5.5/ROUGE-1.5.5.pl', '-e', '/Users/.../files2rouge/pythonrouge/RELEASE-1.5.5/data', '-a', '-m', '-2', '4', '-n', '3', '/var/folders/v4/vzt0mckn5zxgh70vbkx13pb80000gn/T/tmpqxsl34pm/rouge.xml']' returned non-zero exit status 79.
Process Process-8:
Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "files2rouge.py", line 137, in _producer
    q.put([line, ref, summ, get_rouge(ref, summ, score=self.score)])
  File "files2rouge.py", line 35, in get_rouge
    ROUGE_path=ROUGE_path, data_path=ROUGE_data_path)
  File "/Users/.../files2rouge/pythonrouge/pythonrouge.py", line 65, in pythonrouge
    output = subprocess.check_output([ROUGE_path, "-e", data_path, "-a", "-m", "-2", "4","-n", "3", abs_xml_path], stderr=subprocess.STDOUT)
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/Users/.../files2rouge/pythonrouge/RELEASE-1.5.5/ROUGE-1.5.5.pl', '-e', '/Users/.../files2rouge/pythonrouge/RELEASE-1.5.5/data', '-a', '-m', '-2', '4', '-n', '3', '/var/folders/v4/vzt0mckn5zxgh70vbkx13pb80000gn/T/tmplqejfqvb/rouge.xml']' returned non-zero exit status 79.


Evaluated 0 ref/summary pairs in 0.295 seconds (0.000 lines/sec)
/usr/local/lib/python3.6/site-packages/numpy/core/fromnumeric.py:2889: RuntimeWarning: Mean of empty slice.
  out=out, **kwargs)
/usr/local/lib/python3.6/site-packages/numpy/core/_methods.py:80: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
ROUGE-1 (F): nan
ROUGE-2 (F): nan
ROUGE-3 (F): nan
ROUGE-L (F): nan
ROUGE-S4 (F): nan

Any idea?

SyntaxError: invalid syntax for print(*args, **kwargs)

I've got an error as a traceback below:

Traceback (most recent call last):
  File "setup_rouge.py", line 7, in <module>
    from files2rouge import settings
  File "/home/cerdas/files2rouge/files2rouge/__init__.py", line 2, in <module>
    from files2rouge.files2rouge import main
  File "/home/cerdas/files2rouge/files2rouge/files2rouge.py", line 17, in <module>
    from files2rouge import utils
  File "/home/cerdas/files2rouge/files2rouge/utils.py", line 14
    print(*args, **kwargs)
          ^
SyntaxError: invalid syntax

the error occured when i'm trying to run this code:

def tee(saveto, *args, **kwargs):
    """Mimic the tee command, write on both stdout and file
    """
    print(*args, **kwargs)
    if saveto is not None:
        print(file=saveto, *args, **kwargs)

i've read and searching for similar problem, but i still can't solved the error. Thanks

files2rouge tests fail due to not finding installed XML::Parser module

Hi,

I am trying to install files2rouge and encounter some problems with running the tests.

I am using a Perl 5.24.1 installation which is not installed in /usr/. Therefore I have replaced

#!/usr/bin/perl -w

in all of your perl scripts with

#!/usr/bin/env perl

as on our HPC cluster we don't use software installed through the package manager of the operating system

As you can see below, I can use the XML::Parser and @inc contains the paths required for using the XML::Parser package:

[sfux@eu-login-41 ~]$ module list

Currently Loaded Modules:
  1) StdEnv   2) gcc/6.3.0   3) openblas/0.2.20   4) python/3.8.5   5) perl/5.24.1   6) perl-xml-parser/2.44



[sfux@eu-login-41 ~]$ perl -e "use XML::Parser"
[sfux@eu-login-41 ~]$ perl -e "print \"@INC\""
/cluster/apps/gcc-6.3.0/perl-xml-parser-2.44-vex26qd6ng56dplf2gurx522mr45bkmr/lib/perl5/x86_64-linux /cluster/apps/gcc-6.3.0/perl-xml-parser-2.44-vex26qd6ng56dplf2gurx522mr45bkmr/lib/perl5 /cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/perl-5.24.1-ldxzqmuecffm6spae6bclbzzplttud3a/lib/perl5 /cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/perl-5.24.1-ldxzqmuecffm6spae6bclbzzplttud3a/lib/site_perl/5.24.1/x86_64-linux /cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/perl-5.24.1-ldxzqmuecffm6spae6bclbzzplttud3a/lib/site_perl/5.24.1 /cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/perl-5.24.1-ldxzqmuecffm6spae6bclbzzplttud3a/lib/5.24.1/x86_64-linux /cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/perl-5.24.1-ldxzqmuecffm6spae6bclbzzplttud3a/lib/5.24.1 .
[sfux@eu-login-41 ~]$

But when I run the files2rouge tests, then the XML::Parser package is not found:

[sfux@eu-login-41 ~]$ python -m pyrouge.test
2022-03-28 15:14:53,674 [MainThread  ] [INFO ]  Written ROUGE configuration to /tmp/tmpib1iv3gn/rouge_conf.xml
.2022-03-28 15:14:53,763 [MainThread  ] [INFO ]  Processing files in data/SL2003_models_plain_text.
2022-03-28 15:14:53,764 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-01.html.
2022-03-28 15:14:53,764 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-02.html.
2022-03-28 15:14:53,765 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-03.html.
2022-03-28 15:14:53,765 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-04.html.
2022-03-28 15:14:53,766 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-05.html.
2022-03-28 15:14:53,766 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-06.html.
2022-03-28 15:14:53,766 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-07.html.
2022-03-28 15:14:53,767 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-08.html.
2022-03-28 15:14:53,767 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-09.html.
2022-03-28 15:14:53,768 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-10.html.
2022-03-28 15:14:53,768 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-11.html.
2022-03-28 15:14:53,768 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-12.html.
2022-03-28 15:14:53,769 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-13.html.
2022-03-28 15:14:53,769 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-14.html.
2022-03-28 15:14:53,770 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-15.html.
2022-03-28 15:14:53,770 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-16.html.
2022-03-28 15:14:53,770 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-17.html.
2022-03-28 15:14:53,771 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-18.html.
2022-03-28 15:14:53,771 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-19.html.
2022-03-28 15:14:53,772 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-20.html.
2022-03-28 15:14:53,772 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-21.html.
2022-03-28 15:14:53,773 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-22.html.
2022-03-28 15:14:53,773 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-23.html.
2022-03-28 15:14:53,773 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-24.html.
2022-03-28 15:14:53,774 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-25.html.
2022-03-28 15:14:53,774 [MainThread  ] [INFO ]  Saved processed files to /tmp/tmp1ulsphvz.
.2022-03-28 15:14:53,795 [MainThread  ] [INFO ]  Written ROUGE configuration to /tmp/tmp8hsq6f62/rouge_conf.xml
2022-03-28 15:14:53,795 [MainThread  ] [INFO ]  Running ROUGE with command /cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/files2rouge-2.1.0-py3.8.egg/additional_files/ROUGE-1.5.5.pl -e /cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/files2rouge-2.1.0-py3.8.egg/additional_files/data -c 95 -2 -1 -U -r 1000 -n 4 -w 1.2 -a /tmp/tmp8hsq6f62/rouge_conf.xml
Can't locate XML/Parser.pm in @INC (@INC contains: /cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/files2rouge-2.1.0-py3.8.egg/additional_files /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/files2rouge-2.1.0-py3.8.egg/additional_files/XML/DOM.pm line 41.
BEGIN failed--compilation aborted at /cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/files2rouge-2.1.0-py3.8.egg/additional_files/XML/DOM.pm line 70.
Compilation failed in require at /cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/files2rouge-2.1.0-py3.8.egg/additional_files/ROUGE-1.5.5.pl line 177.
BEGIN failed--compilation aborted at /cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/files2rouge-2.1.0-py3.8.egg/additional_files/ROUGE-1.5.5.pl line 177.
E2022-03-28 15:14:54,020 [MainThread  ] [INFO ]  Processing files in data/SL2003_models_rouge_format.
2022-03-28 15:14:54,021 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-01.html.
/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/bin/pyrouge_convert_rouge_format_to_plain_text:14: GuessedAtParserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("lxml"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.

The code that caused this warning is on line 14 of the file /cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/bin/pyrouge_convert_rouge_format_to_plain_text. To get rid of this warning, pass the additional argument 'features="lxml"' to the BeautifulSoup constructor.

  soup = BeautifulSoup(html)
2022-03-28 15:14:54,022 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-02.html.
2022-03-28 15:14:54,023 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-03.html.
2022-03-28 15:14:54,023 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-04.html.
2022-03-28 15:14:54,024 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-05.html.
2022-03-28 15:14:54,025 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-06.html.
2022-03-28 15:14:54,026 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-07.html.
2022-03-28 15:14:54,026 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-08.html.
2022-03-28 15:14:54,027 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-09.html.
2022-03-28 15:14:54,028 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-10.html.
2022-03-28 15:14:54,029 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-11.html.
2022-03-28 15:14:54,029 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-12.html.
2022-03-28 15:14:54,030 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-13.html.
2022-03-28 15:14:54,031 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-14.html.
2022-03-28 15:14:54,032 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-15.html.
2022-03-28 15:14:54,032 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-16.html.
2022-03-28 15:14:54,033 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-17.html.
2022-03-28 15:14:54,034 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-18.html.
2022-03-28 15:14:54,035 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-19.html.
2022-03-28 15:14:54,035 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-20.html.
2022-03-28 15:14:54,036 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-21.html.
2022-03-28 15:14:54,037 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-22.html.
2022-03-28 15:14:54,038 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-23.html.
2022-03-28 15:14:54,038 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-24.html.
2022-03-28 15:14:54,039 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-25.html.
2022-03-28 15:14:54,040 [MainThread  ] [INFO ]  Saved processed files to /tmp/tmpxg0p7eek.
.2022-03-28 15:14:54,147 [MainThread  ] [INFO ]  Writing summaries.
2022-03-28 15:14:54,148 [MainThread  ] [INFO ]  Processing summaries. Saving system files to /tmp/tmpjpsm5yin/system and model files to /tmp/tmpjpsm5yin/model.
2022-03-28 15:14:54,148 [MainThread  ] [INFO ]  Processing files in data/systems_plain.
2022-03-28 15:14:54,148 [MainThread  ] [INFO ]  Processing D30001.M.100.T.A.
2022-03-28 15:14:54,149 [MainThread  ] [INFO ]  Processing D30002.M.100.T.A.
2022-03-28 15:14:54,149 [MainThread  ] [INFO ]  Processing D30003.M.100.T.A.
2022-03-28 15:14:54,150 [MainThread  ] [INFO ]  Processing D30005.M.100.T.A.
2022-03-28 15:14:54,150 [MainThread  ] [INFO ]  Saved processed files to /tmp/tmpjpsm5yin/system.
2022-03-28 15:14:54,150 [MainThread  ] [INFO ]  Processing files in data/models_plain.
2022-03-28 15:14:54,151 [MainThread  ] [INFO ]  Processing D30001.M.100.T.B.
2022-03-28 15:14:54,151 [MainThread  ] [INFO ]  Processing D30001.M.100.T.C.
2022-03-28 15:14:54,152 [MainThread  ] [INFO ]  Processing D30001.M.100.T.D.
2022-03-28 15:14:54,152 [MainThread  ] [INFO ]  Processing D30002.M.100.T.B.
2022-03-28 15:14:54,153 [MainThread  ] [INFO ]  Processing D30002.M.100.T.C.
2022-03-28 15:14:54,153 [MainThread  ] [INFO ]  Processing D30002.M.100.T.E.
2022-03-28 15:14:54,153 [MainThread  ] [INFO ]  Processing D30003.M.100.T.B.
2022-03-28 15:14:54,154 [MainThread  ] [INFO ]  Processing D30003.M.100.T.C.
2022-03-28 15:14:54,154 [MainThread  ] [INFO ]  Processing D30003.M.100.T.F.
2022-03-28 15:14:54,155 [MainThread  ] [INFO ]  Processing D30005.M.100.T.B.
2022-03-28 15:14:54,155 [MainThread  ] [INFO ]  Processing D30005.M.100.T.C.
2022-03-28 15:14:54,155 [MainThread  ] [INFO ]  Processing D30005.M.100.T.G.
2022-03-28 15:14:54,156 [MainThread  ] [INFO ]  Saved processed files to /tmp/tmpjpsm5yin/model.
2022-03-28 15:14:54,157 [MainThread  ] [INFO ]  Written ROUGE configuration to /tmp/tmprks3vxw9/rouge_conf.xml
2022-03-28 15:14:54,157 [MainThread  ] [INFO ]  Running ROUGE with command /cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/files2rouge-2.1.0-py3.8.egg/additional_files/ROUGE-1.5.5.pl -e /cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/files2rouge-2.1.0-py3.8.egg/additional_files/data -c 90 -2 -1 -U -r 1000 -n 2 -w 1.2 -a -m data/config_test2.xml /tmp/tmprks3vxw9/rouge_conf.xml
Can't locate XML/Parser.pm in @INC (@INC contains: /cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/files2rouge-2.1.0-py3.8.egg/additional_files /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/files2rouge-2.1.0-py3.8.egg/additional_files/XML/DOM.pm line 41.
BEGIN failed--compilation aborted at /cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/files2rouge-2.1.0-py3.8.egg/additional_files/XML/DOM.pm line 70.
Compilation failed in require at /cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/files2rouge-2.1.0-py3.8.egg/additional_files/ROUGE-1.5.5.pl line 177.
BEGIN failed--compilation aborted at /cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/files2rouge-2.1.0-py3.8.egg/additional_files/ROUGE-1.5.5.pl line 177.
Traceback (most recent call last):
  File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/bin/pyrouge_evaluate_plain_text_files", line 25, in <module>
    main()
  File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/bin/pyrouge_evaluate_plain_text_files", line 21, in main
    output = rouge.convert_and_evaluate(args.system_id, args.split_sents)
  File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/pyrouge/Rouge155.py", line 368, in convert_and_evaluate
    rouge_output = self.evaluate(system_id, rouge_args)
  File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/pyrouge/Rouge155.py", line 343, in evaluate
    rouge_output = check_output(command, env=env).decode("UTF-8")
  File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/files2rouge-2.1.0-py3.8.egg/additional_files/ROUGE-1.5.5.pl', '-e', '/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/files2rouge-2.1.0-py3.8.egg/additional_files/data', '-c', '90', '-2', '-1', '-U', '-r', '1000', '-n', '2', '-w', '1.2', '-a', '-m', 'data/config_test2.xml', '/tmp/tmprks3vxw9/rouge_conf.xml']' returned non-zero exit status 2.
E.2022-03-28 15:14:54,264 [MainThread  ] [INFO ]  Writing summaries.
2022-03-28 15:14:54,264 [MainThread  ] [INFO ]  Processing summaries. Saving system files to /tmp/tmpwd3ysnud/system and model files to /tmp/tmpwd3ysnud/model.
2022-03-28 15:14:54,264 [MainThread  ] [INFO ]  Processing files in data/systems_plain.
2022-03-28 15:14:54,265 [MainThread  ] [INFO ]  Processing D30001.M.100.T.A.
2022-03-28 15:14:54,265 [MainThread  ] [INFO ]  Processing D30002.M.100.T.A.
2022-03-28 15:14:54,270 [MainThread  ] [INFO ]  Processing D30003.M.100.T.A.
2022-03-28 15:14:54,270 [MainThread  ] [INFO ]  Processing D30005.M.100.T.A.
2022-03-28 15:14:54,271 [MainThread  ] [INFO ]  Saved processed files to /tmp/tmpwd3ysnud/system.
2022-03-28 15:14:54,271 [MainThread  ] [INFO ]  Processing files in data/models_plain.
2022-03-28 15:14:54,271 [MainThread  ] [INFO ]  Processing D30001.M.100.T.B.
2022-03-28 15:14:54,271 [MainThread  ] [INFO ]  Processing D30001.M.100.T.C.
2022-03-28 15:14:54,272 [MainThread  ] [INFO ]  Processing D30001.M.100.T.D.
2022-03-28 15:14:54,272 [MainThread  ] [INFO ]  Processing D30002.M.100.T.B.
2022-03-28 15:14:54,272 [MainThread  ] [INFO ]  Processing D30002.M.100.T.C.
2022-03-28 15:14:54,272 [MainThread  ] [INFO ]  Processing D30002.M.100.T.E.
2022-03-28 15:14:54,273 [MainThread  ] [INFO ]  Processing D30003.M.100.T.B.
2022-03-28 15:14:54,273 [MainThread  ] [INFO ]  Processing D30003.M.100.T.C.
2022-03-28 15:14:54,273 [MainThread  ] [INFO ]  Processing D30003.M.100.T.F.
2022-03-28 15:14:54,274 [MainThread  ] [INFO ]  Processing D30005.M.100.T.B.
2022-03-28 15:14:54,274 [MainThread  ] [INFO ]  Processing D30005.M.100.T.C.
2022-03-28 15:14:54,274 [MainThread  ] [INFO ]  Processing D30005.M.100.T.G.
2022-03-28 15:14:54,274 [MainThread  ] [INFO ]  Saved processed files to /tmp/tmpwd3ysnud/model.
2022-03-28 15:14:54,275 [MainThread  ] [INFO ]  Written ROUGE configuration to /tmp/tmpz7r11ed3/rouge_conf.xml
2022-03-28 15:14:54,275 [MainThread  ] [INFO ]  Running ROUGE with command /cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/files2rouge-2.1.0-py3.8.egg/additional_files/ROUGE-1.5.5.pl -e /cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/files2rouge-2.1.0-py3.8.egg/additional_files/data -c 95 -2 -1 -U -r 1000 -n 4 -w 1.2 -a /tmp/tmpz7r11ed3/rouge_conf.xml
Can't locate XML/Parser.pm in @INC (@INC contains: /cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/files2rouge-2.1.0-py3.8.egg/additional_files /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/files2rouge-2.1.0-py3.8.egg/additional_files/XML/DOM.pm line 41.
BEGIN failed--compilation aborted at /cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/files2rouge-2.1.0-py3.8.egg/additional_files/XML/DOM.pm line 70.
Compilation failed in require at /cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/files2rouge-2.1.0-py3.8.egg/additional_files/ROUGE-1.5.5.pl line 177.
BEGIN failed--compilation aborted at /cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/files2rouge-2.1.0-py3.8.egg/additional_files/ROUGE-1.5.5.pl line 177.
Traceback (most recent call last):
  File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/bin/pyrouge_evaluate_plain_text_files", line 25, in <module>
    main()
  File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/bin/pyrouge_evaluate_plain_text_files", line 21, in main
    output = rouge.convert_and_evaluate(args.system_id, args.split_sents)
  File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/pyrouge/Rouge155.py", line 368, in convert_and_evaluate
    rouge_output = self.evaluate(system_id, rouge_args)
  File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/pyrouge/Rouge155.py", line 343, in evaluate
    rouge_output = check_output(command, env=env).decode("UTF-8")
  File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/files2rouge-2.1.0-py3.8.egg/additional_files/ROUGE-1.5.5.pl', '-e', '/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/files2rouge-2.1.0-py3.8.egg/additional_files/data', '-c', '95', '-2', '-1', '-U', '-r', '1000', '-n', '4', '-w', '1.2', '-a', '/tmp/tmpz7r11ed3/rouge_conf.xml']' returned non-zero exit status 2.
E./tmp/tmpmhimm30m/config_test.xml data/config_test.xml
...
======================================================================
ERROR: test_evaluation (pyrouge.tests.Rouge155_test.PyrougeTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/pyrouge/tests/Rouge155_test.py", line 156, in test_evaluation
    pyrouge_output = rouge.evaluate(system_id=11).strip()
  File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/pyrouge/Rouge155.py", line 343, in evaluate
    rouge_output = check_output(command, env=env).decode("UTF-8")
  File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/files2rouge-2.1.0-py3.8.egg/additional_files/ROUGE-1.5.5.pl', '-e', '/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/files2rouge-2.1.0-py3.8.egg/additional_files/data', '-c', '95', '-2', '-1', '-U', '-r', '1000', '-n', '4', '-w', '1.2', '-a', '/tmp/tmp8hsq6f62/rouge_conf.xml']' returned non-zero exit status 2.

======================================================================
ERROR: test_options (pyrouge.tests.Rouge155_test.PyrougeTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/pyrouge/tests/Rouge155_test.py", line 218, in test_options
    pyrouge_output = check_output_clean(pyrouge_command)
  File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/pyrouge/tests/Rouge155_test.py", line 17, in <lambda>
    check_output_clean = lambda c: check_output(c).decode("UTF-8").strip()
  File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['pyrouge_evaluate_plain_text_files', '-m', 'data/models_plain', '-s', 'data/systems_plain', '-sfp', 'D(\\d+).M.100.T.A', '-mfp', 'D#ID#.M.100.T.[A-Z]', '-id', '1', '-rargs', '"-e /cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/files2rouge-2.1.0-py3.8.egg/additional_files/data -c 90 -2 -1 -U -r 1000 -n 2 -w 1.2 -a -m data/config_test2.xml"']' returned non-zero exit status 1.

======================================================================
ERROR: test_rouge_for_plain_text (pyrouge.tests.Rouge155_test.PyrougeTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/pyrouge/tests/Rouge155_test.py", line 173, in test_rouge_for_plain_text
    pyrouge_output = check_output_clean(pyrouge_command.split())
  File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/pyrouge/tests/Rouge155_test.py", line 17, in <lambda>
    check_output_clean = lambda c: check_output(c).decode("UTF-8").strip()
  File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['pyrouge_evaluate_plain_text_files', '-m', 'data/models_plain', '-s', 'data/systems_plain', '-sfp', 'D(\\d+).M.100.T.A', '-mfp', 'D#ID#.M.100.T.[A-Z]', '-id', '1']' returned non-zero exit status 1.

----------------------------------------------------------------------
Ran 11 tests in 0.761s

FAILED (errors=3)
[sfux@eu-login-41 ~]$

Do you have any idea, why the perl scripts of files2rouge (even after updating the shebang line) don't find the XML::Parser package?

Best regards

Sam

Incorrect Order of Reference and Summary in Readme

I think the current code takes reference as the first argument while the summary is the second, thus README may need to be updated

How to test multi-reference text?

For example, there are 4 references for each article in DUC2004 dataset . How to test the rouge score for this ?

PermissionError: [Errno 13] Permission denied: '/home/Bwwhst28Y/anaconda3/envs/pytorch/lib/python3.6/site-packages/files2rouge-2.1.0-py3.6.egg/files2rouge/ROUGE-1.5.5.pl'

when i run files2rouge /home/Bwwhst28Y/MASS-summarization/tgt.txt /home/Bwwhst28Y/MASS-summarization/hypo.txt
then the error is
PermissionError: [Errno 13] Permission denied: '/home/Bwwhst28Y/anaconda3/envs/pytorch/lib/python3.6/site-packages/files2rouge-2.1.0-py3.6.egg/files2rouge/ROUGE-1.5.5.pl'

How to calculate ROUGE-SU with this package？

I'm a little bit confused about how to calculate ROUGE-SU with this package. The 1.5.5 has this option but I'm not sure how to use it here. Can anyone help?

Thanks

Incorrect L-scores

I am getting incorrect L-scores on the CNN/Daily Mail dataset.
The 1 and 2 scores are correct though.
please help.

fscore does not match with recall and precision

By the f1-score equation,

F1 = 2 * precision * recall / ( precision + recall )

But, the scores calculated by pyrouge.Rouge155() (current calculating system) seem not follow the above equation.

The output example in readme is

1 ROUGE-1 Average_R: 0.28242 (95%-conf.int. 0.25721 - 0.30877)
1 ROUGE-1 Average_P: 0.30157 (95%-conf.int. 0.27114 - 0.33506)
1 ROUGE-1 Average_F: 0.28196 (95%-conf.int. 0.25704 - 0.30722)

1 ROUGE-2 Average_R: 0.10395 (95%-conf.int. 0.08298 - 0.12600)
1 ROUGE-2 Average_P: 0.11458 (95%-conf.int. 0.08873 - 0.14023)
1 ROUGE-2 Average_F: 0.10489 (95%-conf.int. 0.08303 - 0.12741)

1 ROUGE-L Average_R: 0.25231 (95%-conf.int. 0.22709 - 0.27771)
1 ROUGE-L Average_P: 0.26830 (95%-conf.int. 0.23834 - 0.29818)
1 ROUGE-L Average_F: 0.25142 (95%-conf.int. 0.22741 - 0.27533)

However, when I calculate the f1 scores of the example with the above equation, the values are
R1-F1 : 0.29168
R2-F1 : 0.10901
RL-F1 : 0.26006

Where does come this inconsistency?

How much time is needed?

I tried to compare two files with it, but there was no result in one afternoon. The file had 140,000 lines. How long does it take me?

Thanks in advance

ROUGE-1.5.5.pl - XML::Parser dependency error

ROUGE-1.5.5.pl has a XML::Parser dependency, without this module installed, the following error is printed:

Can't locate XML/Parser.pm in @INC (you may need to install the XML::Parser module) (@INC contains: /root/.files2rouge /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.24.1 /usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.24 /usr/share/perl/5.24 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at /root/.files2rouge/XML/DOM.pm line 41.
BEGIN failed--compilation aborted at /root/.files2rouge/XML/DOM.pm line 70.
Compilation failed in require at /root/.files2rouge/ROUGE-1.5.5.pl line 177.
BEGIN failed--compilation aborted at /root/.files2rouge/ROUGE-1.5.5.pl line 177.
Traceback (most recent call last):
  File "/usr/local/bin/files2rouge", line 11, in <module>
    load_entry_point('files2rouge==2.0.0', 'console_scripts', 'files2rouge')()
  File "/usr/local/lib/python3.6/site-packages/files2rouge-2.0.0-py3.6.egg/files2rouge/files2rouge.py", line 81, in main
    output = r.convert_and_evaluate(rouge_args=rouge_args_str)
  File "/usr/local/lib/python3.6/site-packages/pyrouge/Rouge155.py", line 361, in convert_and_evaluate
    rouge_output = self.evaluate(system_id, rouge_args)
  File "/usr/local/lib/python3.6/site-packages/pyrouge/Rouge155.py", line 336, in evaluate
    rouge_output = check_output(command).decode("UTF-8")
  File "/usr/local/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/usr/local/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/root/.files2rouge/ROUGE-1.5.5.pl', '-e', '/root/.files2rouge/data', '-c', '95', '-r', '1000', '-n', '2', '-a', '-m', '/tmp/tmp9qlzfbsc/rouge_conf.xml']' returned non-zero exit status 2.

Probably good to mention in the README

Exception: Could not find any model summaries for the system summary with ID 1000. Specified model filename pattern was: m.[A-Z].#ID#.txt

Hey,

Just aimed to run this command: python files2rouge.py, but encountering the following error:

Preparing documents...
Running ROUGE...
Traceback (most recent call last):
  File "/anaconda/bin/files2rouge", line 11, in <module>
    load_entry_point('files2rouge==2.0.0', 'console_scripts', 'files2rouge')()
  File "/anaconda/lib/python3.5/site-packages/files2rouge-2.0.0-py3.5.egg/files2rouge/files2rouge.py", line 82, in main
    output = r.convert_and_evaluate(rouge_args=rouge_args_str)
  File "/anaconda/lib/python3.5/site-packages/pyrouge/Rouge155.py", line 367, in convert_and_evaluate
    rouge_output = self.evaluate(system_id, rouge_args)
  File "/anaconda/lib/python3.5/site-packages/pyrouge/Rouge155.py", line 334, in evaluate
    self.write_config(system_id=system_id)
  File "/anaconda/lib/python3.5/site-packages/pyrouge/Rouge155.py", line 318, in write_config
    self._config_file, system_id)
  File "/anaconda/lib/python3.5/site-packages/pyrouge/Rouge155.py", line 273, in write_config_static
    id, model_dir, model_filename_pattern)
  File "/anaconda/lib/python3.5/site-packages/pyrouge/Rouge155.py", line 506, in __get_model_filenames_for_id
    "{}".format(id, model_filenames_pattern))
Exception: Could not find any model summaries for the system summary with ID 1000. Specified model filename pattern was: m.[A-Z].#ID#.txt

Is this a common error? How would you solve it?

Thanks,

Check that the file exists or initialize it with 'setup_rouge.py'

Getting the following error when trying to use files2rouge.

Can't load ROUGE settings in '/home/wasiahmad/software/anaconda3/lib/python3.6/site-packages/files2rouge/settings.json'. Check that the file exists or initialize it with 'setup_rouge.py'

How to solve the error?

ROUGE-SU4 is actually ROUGE-S4

An issue coming from pythonrouge, which calls ROUGE-SU4 what is actually a ROUGE-S4 score.

tagucci/pythonrouge#7

Tiny Suggestion: change tempfile.mkdtemp() to TemporaryDirectory()

  File "/home/****/anaconda3/lib/python3.7/site-packages/files2rouge-2.1.0-py3.7.egg/files2rouge/files2rouge.py", line 37, in run
    dirpath = tempfile.mkdtemp()
  File "/home/****/anaconda3/lib/python3.7/tempfile.py", line 366, in mkdtemp
    _os.mkdir(file, 0o700)
OSError: [Errno 28] No space left on device: '/tmp/tmpcch9yqrm'

When Calculating ROUGE score for summary from many checkpoints with a large testset, it creates a huge amount of temporary directories and files in /tmp/.... Seems that these temp files wouldn't be cleaned unless i manual remove or reboot.
So I change it to:

with tempfile.TemporaryDirectory() as dirpath:
    sys_root, model_root = [os.path.join(dirpath, _)
                            for _ in ["system", "model"]]
    ......

Getting error python setup_rouge.py

Hey,

Just aimed to run this command: python setup_rouge.py, but encountering the following error:

Traceback (most recent call last):
File "setup_rouge.py", line 7, in
from files2rouge import settings
File "/Users/ss4164/PycharmProjects/testProject/files2rouge/files2rouge/init.py", line 2, in
from files2rouge.files2rouge import main
File "/Users/ss4164/PycharmProjects/testProject/files2rouge/files2rouge/files2rouge.py", line 17, in
from files2rouge import utils
File "/Users/ss4164/PycharmProjects/testProject/files2rouge/files2rouge/utils.py", line 14
print(*args, **kwargs)
^

Is this a common error? How would you solve it?

Thanks,

TypeError: unexpected keyword argument 'log_level'

hello there,
I just tried to use this, but got the following error messages

Preparing documents...
Running ROUGE...
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.5/bin/files2rouge", line 9, in <module>
    load_entry_point('files2rouge==2.0.0', 'console_scripts', 'files2rouge')()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/files2rouge-2.0.0-py3.5.egg/files2rouge/files2rouge.py", line 65, in main
    log_level=log_level)
TypeError: __init__() got an unexpected keyword argument 'log_level'

Is there any idea to solve this?

Thanks.

Cannot open exception db file for reading

Here is the error message:

Preparing documents... 9 line(s) ignored
Running ROUGE...
Cannot open exception db file for reading: /Users/dongzixuan/.files2rouge/data/WordNet-2.0.exc.db
Traceback (most recent call last):
  File "/Users/dongzixuan/files2rouge/files2rouge/files2rouge.py", line 122, in <module>
    main()
  File "/Users/dongzixuan/files2rouge/files2rouge/files2rouge.py", line 110, in main
    run(args.summary,
  File "/Users/dongzixuan/files2rouge/files2rouge/files2rouge.py", line 75, in run
    output = r.convert_and_evaluate(rouge_args=rouge_args_str)
  File "/usr/local/lib/python3.9/site-packages/pyrouge/Rouge155.py", line 368, in convert_and_evaluate
    rouge_output = self.evaluate(system_id, rouge_args)
  File "/usr/local/lib/python3.9/site-packages/pyrouge/Rouge155.py", line 343, in evaluate
    rouge_output = check_output(command, env=env).decode("UTF-8")
  File "/usr/local/Cellar/[email protected]/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/local/Cellar/[email protected]/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/Users/dongzixuan/.files2rouge/ROUGE-1.5.5.pl', '-e', '/Users/dongzixuan/.files2rouge/data', '-c', '95', '-r', '1000', '-n', '2', '-a', '-m', '/var/folders/9_/m35zxf157c190g94387p8yzw0000gn/T/tmp9ha1v797/rouge_conf.xml']' returned non-zero exit status 79.

I have tried
rm \RELEASE-1.5.5\data\WordNet-2.0.exc.db
and

cd RELEASE-1.5.5\data\
perl WordNet-2.0-Exceptions/buildExeptionDB.pl ./WordNet-2.0-Exceptions ./smart_common_words.txt ./WordNet-2.0.exc.db

but it still show same messeage.
I have really no idea how to fix it, please help me!
And I operate it on MacOS.

THANK YOU SOOOO MUCH!!

Problem with Average_F

Hello, it seems the Average_F is not F1 score, for example

1 ROUGE-1 Average_R: 0.28242 (95%-conf.int. 0.25721 - 0.30877)
1 ROUGE-1 Average_P: 0.30157 (95%-conf.int. 0.27114 - 0.33506)
1 ROUGE-1 Average_F: 0.28196 (95%-conf.int. 0.25704 - 0.30722)

R=0.282, P=0.302, and F1 should be 2*P*R/(P+R)=0.292, is this know's issue or I made some wrong?

Thanks!

pltrdy / files2rouge Goto Github PK

files2rouge's People

Contributors

Stargazers

Watchers

Forkers

files2rouge's Issues

1 ROUGE-1 Average_R: 0.31513 (95%-conf.int. 0.31392 - 0.31628) 1 ROUGE-1 Average_P: 0.41923 (95%-conf.int. 0.41779 - 0.42064) 1 ROUGE-1 Average_F: 0.34849 (95%-conf.int. 0.34722 - 0.34971)

1 ROUGE-2 Average_R: 0.14793 (95%-conf.int. 0.14693 - 0.14890) 1 ROUGE-2 Average_P: 0.19880 (95%-conf.int. 0.19764 - 0.19998) 1 ROUGE-2 Average_F: 0.16334 (95%-conf.int. 0.16228 - 0.16431)

1 ROUGE-L Average_R: 0.29737 (95%-conf.int. 0.29616 - 0.29849) 1 ROUGE-L Average_P: 0.39480 (95%-conf.int. 0.39344 - 0.39618) 1 ROUGE-L Average_F: 0.32844 (95%-conf.int. 0.32725 - 0.32961)

Elapsed time: 19583.610 seconds

Here is the error message:

Thanks for you distribution!

Is this repo got exactly same result with google/seq2seq official repo? Thanks~

Recommend Projects

Recommend Topics

Recommend Org

1 ROUGE-1 Average_R: 0.31513 (95%-conf.int. 0.31392 - 0.31628)
1 ROUGE-1 Average_P: 0.41923 (95%-conf.int. 0.41779 - 0.42064)
1 ROUGE-1 Average_F: 0.34849 (95%-conf.int. 0.34722 - 0.34971)

1 ROUGE-2 Average_R: 0.14793 (95%-conf.int. 0.14693 - 0.14890)
1 ROUGE-2 Average_P: 0.19880 (95%-conf.int. 0.19764 - 0.19998)
1 ROUGE-2 Average_F: 0.16334 (95%-conf.int. 0.16228 - 0.16431)

1 ROUGE-L Average_R: 0.29737 (95%-conf.int. 0.29616 - 0.29849)
1 ROUGE-L Average_P: 0.39480 (95%-conf.int. 0.39344 - 0.39618)
1 ROUGE-L Average_F: 0.32844 (95%-conf.int. 0.32725 - 0.32961)