Tokenizer suggestion about t5-flax-gcp HOT 1 CLOSED

Hi @TinoRed, thanks for reaching out!

First of all, the project is still a work-in-progress but I highly suggest using it5-base instead of the t5-base-it model you mentioned.¹ I will update model cards in the near future with more information, but the it5-base version of the model was trained for much longer and with much more text than its counterpart, so it should produce considerably better performances. Other sizes are also available on my HF Hub page.

The setup you mention is actually the optimal one: the model uses the SentencePieceUnigramTokenizer class only for training, but we save it as a standard T5Tokenizer model and so it is possible to load it using the AutoTokenizer.from_pretrained method, which will be rerouted automatically to T5TokenizerFast.from_pretrained. You can safely ignore warnings.

from t5-flax-gcp.