inte2000 / llm_cpp Goto Github PK
View Code? Open in Web Editor NEWtiktoken is an open-source tokeniser for OpenAI, and TiktokenCpp is a C++ported version. TiktokenCpp using modern C++ language features and providing interface functions that are similar to Tiktoken's Python interface.
License: BSD 3-Clause "New" or "Revised" License