Huge memory cost when initializing the tokenizer about botok HOT 3 CLOSED

I have just tested to build the tries on a syllable basis instead of a char basis, meaning there will be one node per syllable instead of about 4 or 5 of them.
It does significantly reduce de trie size. I can't compare the performance of the tokenizer since it is not meant for such a trie.

It does look like a good improvement, memory usage wise, and the loading time is much better too:

memory_profiler outputs:

Loading Trie... (3s.)
Loading Trie... (2s.)
Loading Trie... (1s.)

Line #    Increment   Line Contents
================================================
    11    691.4 MiB      gmd = get_tok('GMD')
    12    116.7 MiB      pos = get_tok('POS')
    13    186.9 MiB      tsik = get_tok('tsikchen')

The building time is about the same.

Building Trie... (36 s.)
Building Trie... (14 s.)
Building Trie... (7 s.)

from botok.

I did not expect it to consume so much memory. The pickled trie weights about 19Mo for POS profile, so I never thought it would require 600+Mo in memory.
I'll check if there is anything else than the loaded trie that takes so much memory.

from botok.

We have improved tokenizer to work on syllable based Trie (as @drupchen has described above)

from botok.