bmwtsn098 / tokenizer Goto Github PK
View Code? Open in Web Editor NEWThis project forked from opennmt/tokenizer
Fast and customizable text tokenization library with BPE and SentencePiece support
Home Page: https://opennmt.net/
License: MIT License