How to download training data? about opus-mt-train HOT 3 CLOSED

The scripts are still very much hard-coded to our computing environment. I'll try to make them more generic and generally useful as soon as possible. Stay tuned .... OpusTools can definitely be used for retrieving the data. I didn't use opus_express to make the data sets but that would actually be a good way of doing it. So far, I rely on local copies of OPUS that I can retrieve quickly. I can tell you that I also work an improved way of handling language codes and the variation that you can find in OPUS This causes quite some confusion and I hope to improve the situation in some future release.

from opus-mt-train.

Can we use the dataset from https://github.com/Helsinki-NLP/Tatoeba-Challenge ?

from opus-mt-train.

Now the scripts support fetching the data directly from OPUS. The opus-tools python package needs to be installed. Using data from the Tatoeba-Challenge can easily be used by adding the suffix -tatoeba to the build targets. See this for more info: https://github.com/Helsinki-NLP/OPUS-MT-train/blob/master/doc/TatoebaChallenge.md

from opus-mt-train.