sangminwoo / actionmae Goto Github PK

I am working on the reproduction of this paper, and I found that using the baseline model to train the NTU dataset with RGB+Depth two modalities, the highest accuracy I achieved was acc==77, I don’t know why.
In addition, my data set reading speed is very slow.
I wonder if the author or other friends can give me some training experience and suggestions.

I used the dataset downloaded from the official website and used the script to extract frames from the RGB modality videos, 16 frames per sample.
I made very few changes to the source code, and the parameters are basically the default parameters of the source code.

Thank you very much.

Thank you for the interesting work! I just had a few questions from the paper...

In Section 3.2 "Note that only memory token and dummy tokens are affected by the reconstruction loss (i.e., reconstruction loss is not computed for the remaining tokens) during training." Does this mean the reconstruction loss is not backpropagated to the encoders of the "healthy" (retained) modalities? If so, why might this be done/were you able to do experiments ablating this?

Also, I was wondering if you had run experiments on simply dropping out a modality during training for the baseline to help the model get used to missing modalities.

Thank you!