This repository provides the code for Japanese NLI model, a fine-tuned masked language model.
The model showed performance comparable with those reported in JGLUE [Kurihara et al. 2022] and JSICK [Yanaka and Mineshima 2022] papers, in terms of overall accuracy:
Model | JGLUE-JNLI valid [%] | JSICK test [%] |
---|---|---|
[Kurihara et al. 2022] | 91.9 | N/A |
[Yanaka and Mineshima 2022] | N/A | 89.1 |
ours using both JNLI and JSICK | 90.9 | 89.0 |
- Hitomi Yanaka and Koji Mineshima. Compositional Evaluation on Japanese Textual Entailment and Similarity. TACL2022.
- Kentaro Kurihara, Daisuke Kawahara, and Tomohide Shibata. JGLUE: Japanese General Language Understanding Evaluation. LREC2022.
- Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. EMNLP-IJCNLP2019.
- Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised Cross-lingual Representation Learning at Scale. ACL2020.
Yes, we tested only a single run :(
torch.manual_seed(0)
random.seed(0)
np.random.seed(0)
- JSICK
- JGLUE
We converted string label into integer using the following mapping:
label2int = {"contradiction": 0, "entailment": 1, "neutral": 2}
We mimicked batch_size=128
using gradient accumulation 32 * 4 = 128
.
batch_size=32,
shuffle=True,
epochs=3,
accumulation_steps=4,
optimizer_params={'lr': 5e-5},
warmup_steps=math.ceil(0.1 * len(data)),