Representation of Image in EVA about eva HOT 2 CLOSED

This is more about a performance-driven choice.

The MIM pre-trained EVA usually achieves higher accuracy w/ GAP feature in e2e fine-tuning. This is also ture for other MIM pre-trained ViTs, e.g., BEiT, MAE, etc.

For EVA-CLIP, we find using GAP feature achieves sub-optimal performance than using cls_token.

During MIM pre-training, although there is no direct supervision on cls_token, we believe the cls_token indeed represents some kinds of image-level / global content. On the other hand, using GAP image feature is not an absolutely reasonable choice: mask token causes a discrepancy between MIM pre-training and finetuning, as the artificial mask tokens never appear in the fine-tuning stage.

As another example, MAE also uses cls_token for linear probing. We tried with GAP feature for MAE linear probing, the performance is not very satisfactory.

from eva.

OK, thanks for your patient explanation.

from eva.