Confusion about caption tokens and mlm tokens when using mlm about irra HOT 11 CLOSED

Please give a more detailed evidence to prove your claim.

from irra.

The caption_tokens in the diagram are a list that is passed directly to the function, and if the value in the function changes, the value in the list will also change

from irra.

We also confirmed this by printing

from irra.

I have confirmed the issue you mentioned, which was caused by an oversight in my code writing, and I am evaluating the impact of this issue on the final retrieval results.

Anyway, thank you for pointing this out, I will provide more explanation after the evaluation.

from irra.

OK，thanks

from irra.

The two list share the same memory. This may be the reason. Turning to use deepcopy will solve it. However, will it affect the performance?

from irra.

When I changed the shallow copy to a deep copy, i.e. calculating the SDM loss between image feature and masked text feature, there was some performance loss. Here are the results:

And when I calculate the SDM loss bwtween image feature and masked text feature and text feature, the performance loss is lower:

From my point of view, I think the question raised in this issue about using a shallow copy and not a deep copy is equivalent to the fact that I have applied an augmentation to the input text, similar to the data augmentation done to the input image. This leads to a difference in the final result. But it does not affect the conclusions of the paper.
Thank you for your attention to our work, I hope that my response will satisfy you.

from irra.

Is the result of the above table under the setting of id+mlm+sdm？

from irra.

Yes. @lzfff12

from irra.

In the build_random_masked_tokens_and_labels function, there is no deepcopy operation on caption_tokens to get the mlm_tokens. And this will make the two tokens completely same. It means the caption_tokens using for calculating SDM_loss are also masked. But in the paper, it uses unmasked caption_tokens to calculate it. Maybe is it a bug?

from irra.

@Zplusdragon When I was writing the code, I thought the .numpy() function would convert from tensor to a new ndarray, not realising that the two would actually share memory. So, you could say it's a bug.

from irra.