reproducibility problem about mkg_analogy HOT 4 CLOSED

Thanks for your attention!

I will check the results and get back to you soon.

from mkg_analogy.

Hi,

We find that setting g_ea=0 and g_ae=1(equation 4 in paper) helpsft_mart_mkgformer.ckpt most, so you can modify this part (link) to the following code and do test:

# features['sep_idx'] = torch.tensor(sep_idx)
features['attention_mask'] = features['attention_mask'].unsqueeze(1).expand([features['input_ids'].size(0), features['input_ids'].size(1), features['input_ids'].size(1)]).clone()
for i, idx in enumerate(sep_idx):
    features['attention_mask'][i, :idx[2], idx[2]:] = 0

But fine-tuning these two gates can approximate this result, as you said, fine-tuning can reach 0.4001.

from mkg_analogy.

Thanks for your reply.
I've tried the above methods,test more models and read "Issue#9".Maybe I should try more Hyperparameters.
To summarize my experiments:

• the above method did works (hits@10: 0.3289 to 0.3319)but still worse than the results in the paper(0.408).I have summarized three questions.

• In my enviroments,your checkpoints often perform poorly even only half such as "ft_mart_mkgformer.ckpt" ,"ft_mart_mkgformer.ckpt " and "pretrain_vilt.ckpt"
  What's the possible reason?Hardware，library version，datasets or others?

and there are maybe some bugs:

• In "encode_images_data.py", when processor_name = 'vilt', "entity2visual = torch.cat(entity2visual, dim=0)" may cause out of system memory(24GB).
• in "modeling_vilt.py" line 844, when pretraining, " sep_idx += img_length" turns into "None += int"
• in "modeling_vilt.py" line 183,"select" and "patch_index" seems to be stored on different device.
• some scripts and checkpoints seem to be missing.

from mkg_analogy.

hi, have you solved the issues? we have fixed some bugs and you can pull the code.

from mkg_analogy.