lucidrains / zorro-pytorch Goto Github PK

This line is incorrect as it first combines the mask and the token type matrix before comparing with the FUSION token enum value. It can be corrected to this:

zorro_mask = zorro_mask | (token_types_attend_from == TokenTypes.FUSION.value)

Hi, thanks for the work,

I notice some difference in the zorro mask logic to the paper. In the code it goes like:

        # the logic goes
        # every modality, including fusion can attend to self

        zorro_mask = token_types_attend_from == token_types_attend_to

        # fusion can attend to everything

        zorro_mask = zorro_mask | token_types_attend_from == TokenTypes.FUSION.value

        # and both specific modalities like audio and video can attend to fusion

        zorro_mask = zorro_mask | token_types_attend_to == TokenTypes.FUSION.value

But in the paper at page 4, chapter Masked attetion, the author explains:

... our modality-specific representation does not have access to the global representation, ... Specifically, we set $m_{ij}=1$ if $j$ is a part of the fusion representation, otherwise we only set $m_{ij}=1$ if $i$ and $j$ are vectores of the same modality.

Which suggests that the third part of mask isn't needed.
Hope I understood the paper correctly.

Best,
Arthur

Dear @lucidrains ,

Thanks for your amazing contributions to open-source!

I was wondering if you have (pre-)trained Zorro-ViT on any of the large-scale datasets that the paper uses? If yes, would it be possible to release pre-trained checkpoints?

Thanks,
Piyush

Hi, I came across your paper and found it very interesting. It opens up a new direction for exploring multiple modalities. I would love to experiment with your framework. Could you please let me know the ETA on the code release

Hi,

According to the paper (page 4, Chapter Output space, last paragraph), There's a fourth type of query vector called global:

And finally, a global output $o_G$, which attends to all the outputs in the model. Although $o_G$ and $o_F$ do contain similar information, we found it useful to still keep two different heads.

Currently we don't have this type of token in the TokenTypes enum. I'll try to submit a PR for this :)

Thank you so much for your contribution to opensource.

Are you planning to release training code?