Confusion about inconsistent `eos_token_id` about codegen HOT 4 CLOSED

Thanks for pointing this out.

The correct eos token is tokenizer.eos_token_id == 50256.

We will fix the model configuration files.

from codegen.

@xingyaoww Model configs are updated:

In [1]: from transformers import AutoTokenizer, AutoModelForCausalLM
   ...: tokenizer = AutoTokenizer.from_pretrained("Salesforce/codegen-350M-mono")
   ...: model = AutoModelForCausalLM.from_pretrained("Salesforce/codegen-350M-mono")
   ...:
   ...: print(model.config.eos_token_id)
   ...: print(tokenizer.eos_token_id)
Downloading config.json: 100%|██████████████████████████████████████████████████████|
50256
50256

Thanks again for point this out!

from codegen.

@enijkamp , if Eos token id is changed, accuracy on eval dataset will also get impacted, If true then what about paper mentioned accuracy on human eval dataset?

from codegen.

For the HumanEval benchmark execution, the tokenizer is instantiated explicitly, so that the model configuration file has no effect. See,

Thank you for the consideration!

from codegen.