Project website: https://inklab.usc.edu/NumerSense/
Code & Data for EMNLP 2020 paper:
@inproceedings{lin2020numersense,
title={Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Models},
author={Bill Yuchen Lin and Seyeon Lee and Rahul Khanna and Xiang Ren},
booktitle={Proceedings of EMNLP},
year={2020},
note={to appear}
}conda create -n numersense python=3.7
conda activate numersense
# install torch seperately at https://pytorch.org/get-started/locally/ if needed
conda install pytorch==1.6.0 cudatoolkit=10.1 -c pytorch -n numersense
pip install transformers==3.3.1
# pip install happytransformer -U
pip install --editable happy-transformer
pip install tensorboardX
mkdir pred_results
# Optional:
# Install apex following https://github.com/NVIDIA/apex#linux
git clone https://github.com/NVIDIA/apex
cd apex
pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./For masked language models:
python src/mlm_predict.py bert-base \
data/test.core.masked.txt \
results/bert-base.test.core.output.jsonl
python src/mlm_predict.py bert-base \
data/test.all.masked.txt \
results/bert-base.test.all.output.jsonlNote that bert-base can be replaced by any model name in [bert-base, bert-large, roberta-base, roberta-large].
For left-to-right language models:
python src/gpt_predict.py gpt \
data/test.core.masked.txt \
results/gpt.test.core.output.jsonl mkdir saved_models
CUDA_VISIBLE_DEVICES=0 python src/finetune_mlm.py \
--output_dir=saved_models/finetuned_bert_large --overwrite_output_dir \
--model_type=bert \
--model_name_or_path=bert-large-uncased \
--do_train \
--train_data_file=data/gkb_best_filtered.txt \
--do_eval \
--eval_data_file=data/wiki_complete.txt \
--per_gpu_train_batch_size 64 \
--per_gpu_eval_batch_size 64 \
--block_size 64 \
--logging_steps 100 \
--num_train_epochs 3 \
--line_by_line --mlm python src/mlm_infer.py \
reload_bert:saved_models/finetuned_bert_large \
data/test.core.masked.txt \
results/test.core.output.jsonlCheck out data/validation.masked.tsv. We realease 200 annotated examples (132 from the core split and 68 from the all split) for method development so that users can better test their method frquently without submitting the prediction for the test set. Note that these 200 examples should NOT be used for any training. Also, they are still part of the the test data.
To evaluate your model's ability on NumerSense's official test sets,
please submit a prediction file to [email protected], which should contain a json line for each probe example. And a json line should follow the format in the below code snippet. You can also check the example, results/bert-base.test.core.output.jsonl , which is the predictions of BERT-base on core set.
The score key is optional.
When submitting your predictions, please submit both core and all results, and inform us whether you have used the training data for fine-tuning. Thanks!
The evaluation script we will use is src/evaluator.py.
{
"probe": "a bird has <mask> legs.",
"result_list": [
{
"word": "four",
"score": 0.23623309
},
{
"word": "two",
"score": 0.21001829
},
{
"word": "three",
"score": 0.1258428
},
{
"word": "no",
"score": 0.0688955
},
{
"word": "six",
"score": 0.0639159
},
{
"word": "five",
"score": 0.061465383
},
{
"word": "eight",
"score": 0.038915534
},
{
"word": "seven",
"score": 0.014524153
},
{
"word": "ten",
"score": 0.010337788
},
{
"word": "nine",
"score": 0.005654324
},
{
"word": "one",
"score": 1.3131318E-4
},
{
"word": "zero",
"score": 1.10984496E-4
}
]
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent