Sampling after LogitsProcessor ignores scores if previous token was `eos` about transformers HOT 2 CLOSED

@freckletonj hi!

Yes, the eos token is handled differently by a stoppingCriteria and the generation runs in a loop until eos is generated. You can overcome it by removing the eos_token_id from model's config. But in that case you need another stopping criteria, for example (max_new_tokens=30), otherwise the loop will forever.

So a short snippet like this should work and the generation stops exactly after 32nd new token

# set eos tokens to None so that the generation goes on even after eos
model.config.eos_token_id = None
model.generation_config.eos_token_id = None
model.config.forced_eos_token_id = None

stop = StopAfterTokenIsGenerated(tokens['input_ids'].size(1), stop_tokens, sentinel)

output = model.generate(
    **tokens,
    max_new_tokens=32,
    do_sample=False,
    pad_token_id=tokenizer.pad_token_id,
    logits_processor=[stop, ],
)

from transformers.

Ah, that makes sense thank you!

from transformers.