Some questions regarding paper about actionmae HOT 2 CLOSED

Hi, @Divadi
Thanks for your interest in our work!

Q1. Does this mean the reconstruction loss is not backpropagated to the encoders of the "healthy" (retained) modalities?
A1. We do not compute losses for the reconstruction of "healthy" tokens. But this does not mean the reconstruction loss is not backpropagated to "healthy" modality encoders, since the reconstruction output (i.e., ActionMAE output) is conditioned on the "healthy" modality tokens. Rather, only those encoders are learned during training.

Q2. Also, I was wondering if you had run experiments on simply dropping out a modality during training for the baseline to help the model get used to missing modalities.
A2. The baseline model (Transformer fusion) in Table 6 is the same as the Transformer-fusion model in Table 4. We have tried simply dropping the modality in the "inference stage" for the baseline models, but we have not "trained" the baseline models by dropping the modality. This could be an experiment that can confirm the efficacy of ActionMAE. Thank you for the suggestion!

from actionmae.

Ah, I understand; thank you for your clarifications!

from actionmae.