In this project, two datasets for Neural Machine Translation will be explored and compared.

• News Commentary dataset :

The News Commentary Dataset is retrieved from the News Commentary website. The website contains datasets for parallel translated sentences of news in 13 different languages. The dataset can be downloaded here.

• Yandex NLP dataset :

Yandex offers a dataset for parallel sentences in Russian and English. This dataset outsizes the News Commentary dataset in sentence count.

The dataset contains 280984 pairs of sentences; each pair contains two sentences: The English and the Russian versions of the sentence.

Some samples from the dataset:

Yandex dataset contains one million pairwise English-Russian sentences. The Yandex dataset contains more data points as well as it covers more topics in its sentences.

Some samples from the dataset:

for more analysis about the datasets please refer to data_exploration/news_commentary_exploration.ipynb and data_exploration/yandex_data_exploration.ipynb

More details will be added later on in the project