compat with transformers >= 4.40 and tokenizers >= 0.19 about transformers.js HOT 2 OPEN

Thanks for reporting! Do you have an example model on the HF hub (and example input text) for which transformers.js now produces an incorrect tokenization? That would help a ton!

from transformers.js.

I've been using "qanastek/pos-french-camembert" and almost any phrase was failing when I converted the model to onnx with the newer transformers version. One short phrase was "J'ai fini". With the pre-19 tokenizers, you get the correct parts of speech

  ("J" "PPER1S")
  ("'", "ADV")
  ("ai", "AUX")
  ("fini", "VPPMS")

With the latest, you get all the entities as MOTINC, which is "unknown words" https://github.com/qanastek/ANTILLES/blob/a860b50a31be9fe81f4b20a1914e84a4f4c7e910/README.md#new-tags-based-on-lia_tagg.

If you need more information, I can make a quick repro repo.

from transformers.js.