We provide a general framework for training subword-informed word representations by varying the following components:

• subword segmentation methods;
• subword embeddings and position embeddings;
• composition functions;

For the whole framework architecture and more details, please refer to the reference.

There are 4 segmentation methods, 3 possible ways of embedding subwords, 3 ways of enhancing with position embeddings, and 3 different composition functions.

Here is a full table of different options and their labels:

For example, sms.wwppmtxatt means we use CHIPMUNK as segmentation, insert word token into the subword sequence, enhance with additive position embedding, and use multi-head self-attention as composition function.

Taking the word dishonestly as an example, with different segmentation methods, the word will be segmented into the following subword sequence:

• ChipMunk: (<dis, honest, ly>) + (PREFIX, ROOT, SUFFIX)
• Morfessor: (<dishonest, ly>)
• BPE (10k merge ops): (<dish, on, est, ly>)
• Character n-gram (from 3 to 6): (<di, dis, ... , ly>, <dis, ... ,tly>, <dish, ... , stly>, <disho, ... , estly>)

where < and > are word start and end markers.

After the segmentation, we will obtain a subword sequence S for each segmentation method, and another morphortactic tag sequence T for sms.

We can embed the subword sequence S directly into subword embedding sequence by looking up in the subword embedding matrix, or insert a word token (ww) into S before embedding, i.e. for sms it will be (<dis, honest, ly>, <dishonestly>).

Then we can enhance the subword embeddings with additive (pp) or elementwise (mp) multiplication.

For sms, we can also embed the concatenation of the subword and its morphortactic tags (wp): (<dis:PREFIX, honest:ROOT, ly>:SUFFIX). And <dishonest>:WORD will be inserted if we choose ww. Note that position embeddings are not applicable to wp as a kind of morphological position information has already been provided.

• Make sure you are in subword_study folder
• Run ./prereq.sh file
  • if file is not executing, run chmod +x prereq.sh and run above command again (Repeat this step for all the .sh files)
• After this command the following files for Marathi should be generate in the 'toy_data/ma' folder:
  • ma.sent.1m
  • ma.sent.1m.5.word
  • ma.sent.1m.5.dict
  • ma.sent.1m.5.morf
  • ma.sent.1m.5.bpe

• cd code
• ./run.sh ma encodingType config learningRate batch_size
• the respective trained model will be generated in the ./code/outfiles/ folder

• cd eval
• run ./eval.sh (it will take a while to execute)
• result plot image and html should be generated in the present working directory

• A Systematic Study of Leveraging Subword Information for Learning Word Representations. Yi Zhu, Ivan Vulić, and Anna Korhonen. In Proc. of NAACL 2019.