MuLD (Multitask Long Document Benchmark) is a set of 6 NLP tasks where the inputs consist of at least 10,000 words. The benchmark covers a wide variety of task types including translation, summarization, question answering, and classification. Additionally there is a range of output lengths from a single word classification label all the way up to an output longer than the input text.
This repo contains official code for the paper MuLD: The Multitask Long Document Benchmark.
The easiest method is to use the Huggingface Datasets library:
import datasets
ds = datasets.load_dataset("ghomasHudson/muld", "NarrativeQA")
ds = datasets.load_dataset("ghomasHudson/muld", "HotpotQA")
ds = datasets.load_dataset("ghomasHudson/muld", "Character Archetype Classification")
ds = datasets.load_dataset("ghomasHudson/muld", "OpenSubtitles")
ds = datasets.load_dataset("ghomasHudson/muld", "AO3 Style Change Detection")
ds = datasets.load_dataset("ghomasHudson/muld", "VLSP")Or by cloning this repo:
import datasets
ds = datasets.load_dataset("./muld.py", "NarrativeQA")
...If you prefer to download the data files yourself:
- NarrativeQA Train Val Test
- HotpotQA Train, Val
- Character Archetype Classification Train Val Test
- OpenSubtitles Train Test
- Style Change Train Val Test,
- VLSP Test
- Mirror: Test
If you use our benchmark please cite the paper:
@InProceedings{hudson-almoubayed:2022:LREC,
author = {Hudson, George and Al Moubayed, Noura},
title = {MuLD: The Multitask Long Document Benchmark},
booktitle = {Proceedings of the Language Resources and Evaluation Conference},
month = {June},
year = {2022},
address = {Marseille, France},
publisher = {European Language Resources Association},
pages = {3675--3685},
url = {https://aclanthology.org/2022.lrec-1.392}
}
Additionally please cite the datasets we used (particularly NarrativeQA, HotpotQA, and Opensubtitles where we directly use their data with limited filtering).
The following table is necessary for this dataset to be indexed by search engines such as Google Dataset Search.
| property | value | ||||||
|---|---|---|---|---|---|---|---|
| name | MuLD |
||||||
| alternateName | Multitask Long Document Benchmark |
||||||
| url | https://github.com/ghomasHudson/muld |
||||||
| description | MuLD (Multitask Long Document Benchmark) is a set of 6 NLP tasks
where the inputs consist of at least 10,000 words. The benchmark covers a
wide variety of task types including translation, summarization,
question answering, and classification. Additionally there is a range of
output lengths from a single word classification label all the way up
to an output longer than the input text. |
||||||
| citation | https://arxiv.org/abs/2202.07362 |
||||||
| creator |
|
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent