get_encoding error for gpt2, but other encodings fine about tiktoken HOT 5 CLOSED

The logic is here:

So typically python -c 'import tempfile; import os; print(os.path.join(tempfile.gettempdir(), "data-gym-cache"))'

from tiktoken.

Hm, thanks for the detailed environment information, but I'm not able to reproduce.

Can you set export TIKTOKEN_CACHE_DIR="" and retry? This environment variable will prevent tiktoken from using a cache for the vocab files it downloads.

Note that even in the simple publicly available tests this code path is tested:

from tiktoken.

I tried to set the key, but not solved. Is there a specific path for the cache? I might need to delete the cache manually.

from tiktoken.

If that doesn't help, maybe you could set a breakpoint and see what the difference between those two dictionaries is.

from tiktoken.

Woo, that works, after deleted the cached files, it turns right now. Thanks a lot!

There might be an error of the file during or after downloading. Not sure if it is needed to check the cached file before use it, or in that assert bpe_ranks == encoder_json_loaded line, might print more info if it failed.

from tiktoken.