Unknown PostProcessor type: Sequence about transformers.js HOT 9 CLOSED

Will most likely do one tomorrow. Just finalizing a few other things. :)

from transformers.js.

@xenova Any thoughts on this? This is preventing loading llama 3 8b, which is a bummer.

from transformers.js.

Hi there 👋 Thanks for the report!

Luckily, we already support the ByteLevel and TemplateProcessing post-processors, so the only thing needed is to implement the Sequence post-processor.

Similarly, we already support sequences of normalizers, decoders, and pre-tokenizers, and a similar pattern can be adapted for post-processors. Is this something you'd be interested in adding? If so, I'd be happy to review a PR.

from transformers.js.

Sorry I don't plan to work on this issue, I'm just reporting a random issue I met.

from transformers.js.

No worries! It's super simple, so I'll add it soon. Thanks again for reporting!

from transformers.js.

@xenova I am also encountering this issue. I was going to take a pass at it, but I don't understand the internals well enough to understand how to meaningfully accumulate the token_type_ids generated by post processors (ref).

from transformers.js.

Here's the rust code for it: https://github.com/huggingface/tokenizers/blob/25aee8b88c8de3c5a52e2f9cb6281d6df00ad516/tokenizers/src/processors/sequence.rs#L18-L36 and it should be easy to translate into JS.

from transformers.js.

I added support for it in #771. See here for example usage.

from transformers.js.

@xenova 💯 thank you, that is awesome. What cadence are you doing releases on?

from transformers.js.