how to retain spiece token markers about transformers.js HOT 2 OPEN

I should add that I the model I'm using is the result of calling python -m scripts.convert --quantize --model_id qanastek/pos-french-camembert

from transformers.js.

I read through the tokenizer code and found that nulling out the tokenizer's decoder tokenizer.decoder = null allows passing through the tokens due to this check: https://github.com/xenova/transformers.js/blob/main/src/tokenizers.js#L2997.

It seems that the code works correctly, as it's parsing the decoder and applying it

   "decoder": {
      "type": "Metaspace",
      "replacement": "▁",
      "add_prefix_space": true,
      "prepend_scheme": "always"
    },

But when playing around with this, it seems to me that the decoder should be applied to a list of tokens, not to a single token: When I null out the tokenizer's decoder and run the decoder on the full list of tokens, I recover the original input.

from transformers.js.