Question again ：） about grounding_llms_with_online_rl HOT 3 CLOSED

Hi,

The algorithm actually updates both the value and the policy. You can find this update in the PPOUpdater looking at at PPO's loss: https://github.com/flowersteam/Grounding_LLMs_with_online_RL/blob/276aab720d0c5e9ba8e39137dfdf9ae4c38b5c98/experiments/train_language_agent.py#LL264C13-L264C107

In our paper and in the code of this repo, the LLM isn't frozen, meaning that performing an update to minimize this loss will affect the whole model (i.e. the LLM and the value head). So as you said, updating the LLM (the whole network in our case) is needed for policy improvement.

As a piece of additional information, we also performed some very small tests freezing the LLM and training only the Language Modeling heads and value head, which resulted in poor results. One can however avoid finetuning the whole LLM by using a lightweight method such as LoRA: https://twitter.com/ClementRomac/status/1667120630762962945.

Hope this helps, do not hesitate to ask further questions :)

from grounding_llms_with_online_rl.

@ClementRomac

from grounding_llms_with_online_rl.

Okay, I thought the green part of the algorithm is frozen 😅

from grounding_llms_with_online_rl.