TRL orpo gives everything Nan about trl HOT 6 OPEN

I'm experiencing same issue :( seems like the grad_norm suddenly diverges to infinity after some iterations.

from trl.

@gagan3012 @hbin0701 do you see this with some specific dataset? Here is my run of orpo.py:

https://wandb.ai/krasul/huggingface/runs/rqu2awe3?nw=nwuserkrasul

using:

python examples/scripts/orpo.py \
    --model_name_or_path=gpt2 \
    --per_device_train_batch_size 4 \
    --max_steps 1000 \
    --learning_rate 1e-3 \
    --gradient_accumulation_steps 1 \
    --logging_steps 10 \
    --eval_steps 500 \
    --output_dir="orpo_anthropic_hh" \
    --optim rmsprop \
    --warmup_steps 150 \
    --report_to wandb \
    --bf16 \
    --logging_first_step \
    --no_remove_unused_columns \
    --use_peft \
    --lora_r=16 \
    --lora_alpha=16

from trl.

I was using mistral 0.2

from trl.

Hello @gagan3012, I just saw this issue and would like to add some comments!

Although I do not know the specific environment or dataset you are using, it is generally recommended that you use a lower learning rate and beta for larger models.

For example, this code for reproducing kaist-ai/mistral-orpo-capybara-7k uses a maximum learning rate of 5e-6 and beta of 0.05. (this code is not for TRL ORPOTrainer, by the way)

accelerate launch --config_file ./src/accelerate/fsdp.yaml main.py \
    --lr 5e-6 \
    --torch_compile False \
    --beta 0.05 \
    --lr_scheduler_type inverse_sqrt \
    --warmup_steps 100 \
    --model_name mistralai/Mistral-7B-v0.1 \
    --data_name argilla/distilabel-capybara-dpo-7k-binarized \
    --num_train_epochs 3 \
    --optim adamw_bnb_8bit \
    --gradient_accumulation_steps 1 \
    --prompt_max_length 1792 \
    --response_max_length 2048 \
    --per_device_train_batch_size 8 \
    --per_device_eval_batch_size 8 \
    --num_proc 8 \
    --flash_attention_2

I am not sure which dataset you are training to, but I would start with a beta of 0.1 and a learning rate of 5e-6 for the first.
I will add some general guidelines for selecting the learning rate and beta by model size/dataset style in this repo by this week!

from trl.

Hello,
When using the Orpo repo, i don't face this issue, but I face this issue when I use TRL, which is very puzzling

from trl.

Is your prompt preparation correct?

TRL expects the "chosen" and "rejected" columns to be a) formatted (but not tokenized) and b) to EXCLUDE the prompt.

TRL also does not add any bos or eos tokens, so you need to do that in the chat_template. Further, since you'll be formatting chosen and rejected columns without the prompt, you need to ensure that the bos is NOT included there...

from trl.