This is a data/code release accompanying this paper:

• Title: "A Pragmatics-Centered Evaluation Framework for Natural Language Understanding"
• Authors: Damien Sileo, Tim Van de Cruys, Camille Pradel and Philippe Muller
• Accepted at LREC2022
• https://arxiv.org/abs/1907.08672

PragmEval is a compilation of 11 evaluation datasets with a focus on discourse, that can be used for evaluation of English Natural Language Understanding, or as auxiliary training tasks for NLP models.

While the idea of meaning as use permeates NLP, it's not clear that current evaluations fully account for that aspect. Previous evaluation frameworks have no clear way to evaluate how models deal with implicatures or different kinds of speech acts, and are arguably focussing on semantics (Natural Language Inference of Semantic Similarity) rather than use. We propose a discourse-centered evaluation with a focus on meaning as use.

from datasets import load_dataset
dataset = load_dataset('pragmeval','gum')

Evaluate your model with this script: https://colab.research.google.com/drive/1sg--LF4z7XR1wxAOfp0-3d4J6kQ9nj_A?usp=sharing

git clone https://github.com/synapse-developpement/PragmEval.git

The preprocessed datasets are available in the pragmeval folder in tsv format.

Run the bash get_data.bash in data to download dataset from the sources

Run the notebook Make PragmEval 1.0 after having specified pragmeval_base_path in the third cell to perform preprocessing and exports.

@inproceedings{sileo-etal-2022-pragmatics,
    title = "A Pragmatics-Centered Evaluation Framework for Natural Language Understanding",
    author = "Sileo, Damien  and
      Muller, Philippe  and
      Van de Cruys, Tim  and
      Pradel, Camille",
    booktitle = "Proceedings of the Thirteenth Language Resources and Evaluation Conference",
    month = jun,
    year = "2022",
    address = "Marseille, France",
    publisher = "European Language Resources Association",
    url = "https://aclanthology.org/2022.lrec-1.255",
    pages = "2382--2394",
}

For further information, you can contact:

damien dot sileo at gmail dot com