bert-base-turkish-qa's Introduction

BERT-base-Turkish-QA

A Turkish question answering system made by fine-tuning BERTurk, which is a BERT base model transformer. We have trained and evaluated the exact match and F1 scores using different Turkish data sets, then compared the evaluation results. In our final model, we have concatenated all of the Turkish data sets into one data set and trained our model using the whole training data set. You can check out our final models Turkish BERTurk Based Model, Turkish XLM-R Based Model.

This project is made during our joint internship at SESTEK Speech Enabled Software Technologies.

Data Sets

OkanVK's Turkish Reading Comprehension Question Answering Data Set

TQuAD (Turkish Question Answering Data Set)

XQuAD (Cross-lingual Question Answering Data Set)

Kuzgunlar's Data Set

Base Models

BERTurk

XLM-R

Model Comparison

Base Models	Training Set	Evalulation Set	Results		Hyperparameters

			Exact	F1	epoch	max_seq_length	doc_stride	learning_rate
best-base-turkish-128k-cased	whole_train_dataset	whole_dev_dataset	62.48	81.60	2	512	128	3,00E-05
	whole_train_dataset	okanvk_dev	62.48	81.66	2	512	128	3,00E-05
	whole_train_dataset	tquad_dev	62.22	80.42	2	512	128	3,00E-05
	whole_train_dataset	xquad.tr	45.89	66.37	2	512	128	3,00E-05
best-base-turkish-128k-cased	Tquad_train	whole_dev_dataset	56.32	76.86	2	512	128	3,00E-05
	Tquad_train	okanvk_dev	56.31	76.87	2	512	128	3,00E-05
	Tquad_train	tquad_dev	57.40	78.68	2	512	128	3,00E-05
	Tquad_train	xquad.tr	41.76	60.83	2	512	128	3,00E-05
best-base-turkish-128k-cased	Okanvk_train	whole_dev_dataset	60.37	80.53	2	512	128	3,00E-05
	Okanvk_train	okanvk_dev	60.37	80.63	2	512	128	3,00E-05
	Okanvk_train	tquad_dev	58.63	78.43	2	512	128	3,00E-05
	Okanvk_train	xquad.tr	46.38	70.74	2	512	128	3,00E-05
XLM-R Base	whole_train_dataset	whole_dev_dataset	54.60	76.83	2	512	128	3,00E-05
	whole_train_dataset	okanvk_dev	54.68	76.94	2	512	128	3,00E-05
	whole_train_dataset	tquad_dev	53.48	75.26	2	512	128	3,00E-05
	whole_train_dataset	xquad.tr	42.27	61.72	2	512	128	3,00E-05
Savasy QA (not base)	Tquad_train	tquad_dev	62.56	80.48