The project targets on question answering models for 2020 Formosa Grand Challenge. It provides tools for correctness evaluation, analytics and visualization.
The project only support Python3. You need to install following packages via pip or conda:
- json-lines
- pandas
- tabulate
Run following command and you will see the results:
python3 eval.py \
./examples/official_1_questions.json \
./examples/official_1_answers.json \
./examples/predictions.jsonl
Run following command to see detail usages:
python3 eval.py -h
The usages is shown as follows:
usage: eval.py [-h] [-o OUTPUT]
[-f {plain,simple,github,grid,fancy_grid,pipe,orgtbl,jira,presto,psql,rst,mediawiki,moinmoin,youtrack,html,latex,latex_raw,latex_booktabs,textile}]
questions_path answers_path predictions_path
Evaluation tool of question answering model for 2020 Formosa Grand Challenge
positional arguments:
questions_path Path to questions JSON file (.json)
answers_path Path to ground truth answers JSON file (.json)
predictions_path Path to predictions JSON Lines file (.jsonl)
optional arguments:
-h, --help show this help message and exit
-o OUTPUT, --output OUTPUT
Path to directory that saves the results (default:
output)
-f {plain,simple,github,grid,fancy_grid,pipe,orgtbl,jira,presto,psql,rst,mediawiki,moinmoin,youtrack,html,latex,latex_raw,latex_booktabs,textile}, --format {plain,simple,github,grid,fancy_grid,pipe,orgtbl,jira,presto,psql,rst,mediawiki,moinmoin,youtrack,html,latex,latex_raw,latex_booktabs,textile}
Format to print the result tables (default: github)
After running the tools, you will get three kinds of result: details, correctness, error reasons and scores.
The result shows detail for each of your model prediction. You can see a compact version report on stdout and full version report in output/details.csv
.
Meaning of the fields:
PID
: Passage IDQID
: Question IDPassage
: Passage text (only in full version)Question
: Question text (only in full version)Question Type
: Question type, it can be:passage_span
: The answer should be extracted from passagequestion_span
: The answer should be extracted from questionmultiple_spans
: The answer should be extracted from passage (more than one span)yesno
: The answer should be yes or nomath
: The answer is a math problem (arithmetic or counting)
Answer
: Ground truth text (only in full version)Prediction
: Your model prediction text (only in full version)Prediction Type
: Your model prediction typeCorrect?
: Your model prediction is correct or notScore
: The score your model got for this questionNon-Correct Reason
: If your model prediction is not correct, what is the reason? It can be:right_type_wrong_prediction
: Your model predicted a correct question type, but its prediction is wrongwrong_type
: Your model predicted a wrong question type
An example of compact version details report is shown as follows:
The result shows correctness count and rate of each question type. You can see the report on stdout and in output/correctness.csv
.
Meaning of the fields:
Type
: Question type, it can be:Question Count
: Total count of that question typepassage_span
: The answer should be extracted from passagequestion_span
: The answer should be extracted from questionmultiple_spans
: The answer should be extracted from passage (more than one span)yesno
: The answer should be yes or nooverall
: All question typesmath
: The answer is a math problem (arithmetic or counting)
Correct Count
: Correct count of that question type from your predictionsCorrect Rate
: Correct rate of that question type from your predictions
An example is shown as follows:
The result shows count and rate for each reason for non-correct predictions. You can see the report on stdout and in output/error_reasons.csv
.
Meaning of the fields:
Reason
: The reason that your model predictions are not correct. It can be:right_type_wrong_prediction
: Your model predicted a correct question type, but its prediction is wrongwrong_type
: Your model predicted a wrong question type
Count
: Count of that reason from your non-correct predictionsCount
: Rate of that reason from your non-correct predictions
An example is shown as follows:
You will see the total your model got.