camtl / ca-mtl Goto Github PK

In your paper, you mentioned that this model frozen parameters in the 1-6 layers. However, in your code, the parameters in first six layers implemented by BertLayer9 seems not been frozen.

It looks like there are arguments (percent_of_max_data_size & uniform_mt_sampling) missing that are consumed by the MultiTaskTrainer for the __len__ method.

This is causing errors in some of the super methods.
https://github.com/huggingface/transformers/blob/b42586ea560a20dcadb78472a6b4596f579e9043/src/transformers/trainer.py#L367

I've traced them through the repo but found only this reference and do not see them listed in the cl args.

How would you suggest mitigating?

Hi there,

Thanks for sharing the code. I am pretty excited to try the code. However, the performance cannot reproduce the paper results.

I ran the code with command:
python run.py --model_name_or_path CA-MTL-base --data_dir $DATA_DIR --output_dir $OUTPUT_DIR --do_train --do_eval

The results for
cola is 0.00286100001416597
mnli is 0.39949057564951607
rte is 0.4657039711191336
qqp is 0.6150630719762552
sst-2 is 0.6502293577981652
qnli is 0.6637378729635731

This result is a complete failure. Could you provide the command to reproduce your results in paper?

Should change

python run.py --model_name_of_path CA-MTL-base --data_dir $DATA_DIR --output_dir $OUTPUT_DIR --do_train

change model_name_of_path to model_name_or_path

Random Joe looking at your paper (very interesting work by the way)

I was looking at the code and this line struck me. It's just mapping an index to itself. This attribute self.task_id_2_task_idx is used only here (idem for BERT-base) which ends up doing nothing when we compare task_type before and after.

All of this to get embedding indexes. Just wanted to raise your attention in case that was not the expected behaviour.

Hi. I am working to adapt your code to another application and am running into an error here:

Package versions align with requirements.txt

Any suggestions?