- To create vocabulary and embeddings run
$ python3 create_vocab.py <embedding_size> <num_iterations>
Where embedding_size is the size (dimensions) of embeddings to be generated (eg - 128), num_iterations is the number of iterations to be run on dictionary to generate embeddings (eg - 10)
This will create two files - vocab.txt and embeddings (pickle dump)
- To test the tokenizer run
$ python3 run_tokenizer.py <path/to/file/to/tokenize>
This will print each line and its tokenization on screen, redirect it to a file for better readability
- Dictionary -
https://github.com/tusharlock10/Dictionary/blob/master/data.7z
- Lemma list -
https://github.com/skywind3000/lemma.en
- Suffixes -
https://en.wiktionary.org/wiki/Appendix:English_suffixes
- Prefix -
https://en.wikipedia.org/wiki/Prefix