Substitus is a computer program for linguistics-based word decomposition. The name is an abbreviation of the phrase substitutive segmenter which references to the used “method of squares” proposed by Joseph Harold Greenberg and independently reinvented and further developed while writing this work. Primarily, the application is designed to segment words (in arbitrary language), but it can handle any sequences. All it needs to work is a frequency dictionary of the processed language (100,000 word forms or more). You will find more details in the thesis (see the official archive).

Substitus is implemented in Java 8 as a console application. It is distributed in a single JAR file (less than 1 MB), with no library dependencies.

$ echo "strawberry" | java -jar substitus.jar segmentize-words \
    --frequency-list english.fwl \
    --output-format html \
    --verbosity 1

Segmentation s trawberry:   0.00% ... no evidence
Segmentation s trawberry:   0.00% ... no evidence
Segmentation st rawberry:   0.00% ... no evidence
Segmentation str awberry:  15.65% ... [st-] × [-uck, -eaming, -eam]
Segmentation stra wberry:   2.29% ... [co-, cro-, ro-] × [-tton, -ps, -pped]
Segmentation straw berry:  58.05% ... [black-, blue-, gold-] × [-ford, -man, -son]
Segmentation strawb erry:   1.79% ... [thi-, m-, v-] × [-ale, -oss, -ridge]
Segmentation strawbe rry:   0.32% ... [so-, sha-, co-] × [-ar]
Segmentation strawber ry:   7.72% ... [count-, ma-, ga-] × [-y]
Segmentation strawberr y:  95.10% ... [universit-, part-, compan-] × [-ies, -ies., -ries]

s 0.00 t 0.00 r 0.16 a 0.02 w 0.58 b 0.02 e 0.00 r 0.08 r 0.95 y
straw berr y

The output consists of (optional) segmentation details and two results, probabilistic and binarised. The binarised version is just a simple modification of the probabilistic one.

In order to learn grammar, Substitus needs a frequency word list. The file consist of pairs frequency – word. The columns may be divided by either spaces or tabs.

582770  the
268766  to
256531  and
…
1169    attention
1168    45
1166    focus
…
1       university-educated
1       W.K.
1       Zhuan

Substitus settings:

• frequency word list: two million words, web-corpus-based
• split function: segmentability ≥ 0.5 || (segmentability > 0 && length (sequence) > 8)

Dictionary: Retrográdní morfematický slovník češtiny (1975)

Dictionary: Morfematický slovník slovenčiny (1999)

Here, productivity is defined as the number of word forms the token appears in, regardless their frequency. That’s why you will not find tokens like “but” here.

For each language, the tokens were extracted from a large web-corpus-based word list of one million most frequent word forms. The examples, shown in tooltips, are sometimes bad (especially for short tokens), but the tokens themselves should be fine.