[Bug] Helsinki Multilingual models errors when using required >>id<< tokens about transformers.js HOT 6 CLOSED

Awesome! Closing now 😄

from transformers.js.

12163dd should fix this 👍 Let me know if it works!

Here's some example code:

const pipe = await pipeline('translation', 'Helsinki-NLP/opus-mt-en-mul');
const result = await pipe(">>jpn<< I love pizza", {
    do_sample: false,
    num_beams: 1
});
// Outputs: [{translation_text: 'ピザを愛してる'}]

I will close the issue once I publish a full release.

from transformers.js.

You're right! I didn't account for that, my bad! Let me work on that.

Here's the part of the code I missed: https://github.com/huggingface/transformers/blob/12d51db243a00726a548a43cc333390ebae731e3/src/transformers/models/marian/tokenization_marian.py#L204-L213

Good catch!

12163dd indeed fixes the issue, thank you!

from transformers.js.

You're right! I didn't account for that, my bad! Let me work on that.

Here's the part of the code I missed:
https://github.com/huggingface/transformers/blob/12d51db243a00726a548a43cc333390ebae731e3/src/transformers/models/marian/tokenization_marian.py#L204-L213

from transformers.js.

Could this be related to #66 ?

from transformers.js.

Could this be related to #66 ?

Although the error message is the same, I think that's just a general error that onnxruntime throws when something goes wrong :/ That's the reason I added the extra logging statements so that we can perhaps see what the actual issue is.

One thing that might be causing it is an out-of-memory issue which can happen if too many beams are being run (which happens since the default config, which comes from the python library, has num_beams set to something > 1). I'll have to look into it a bit more though.

from transformers.js.