dareylyon / spm_precompiled Goto Github PK
View Code? Open in Web Editor NEWThis project forked from huggingface/spm_precompiled
Highly specialized crate to parse and use `google/sentencepiece` 's precompiled_charsmap in `tokenizers`
License: Apache License 2.0