jihoontack / mac Goto Github PK

as title.

Thanks for your great work and I am a new guy in this research area. I try to follow your code and one question confuses me. In the common/dataset.py, we just concat the question and answer pair to form the input_ids and attention_masks.

And in the models/amortized_enddec.py, we input the "qa_ids" and "qa_attention" into base_lm for the "qa_loss".

I wonder if this will cause information leak to the llm, since we already input the answer with the question to it? As I mentioned I am new in this area and not familiar with the whole online adaptation pipeline. Would you kindly please give me some ideas? Looking forward to your reply.