the instruction tuning stage about salmonn HOT 2 CLOSED

In fact, the model that has only gone through the pre-training stage can only perform ASR and AAC tasks and is completely incapable of performing any zero-shot tasks. In other words, the problem of task overfitting is even more serious at this point. As you can see in the paper, we introduce a large amount of QA data in the instruction tuning stage, so that the model can see more abundant prompts, thus alleviating the situation of the model not following instructions. However, the model is still struggling to do more difficult tasks, without reducing the lora factor or being activated.

For your second question, I don't quite understand. In pre-training stage and instruction tuning stage, the Q-Former and LoRA are both updated. We used the model after instruction tuning to plot Figure 3. I think the phenomenon your mentioned can only demonstrate that reducing lora scaling to 2.0 is sufficient to activate the model capacity, but does not directly indicate that the pre-trained model can solve these tasks.

from salmonn.

I will close this issue. If you have any question, welcome to reopen it.

from salmonn.