Model Architecture For Fine-tuning about vit-pytorch HOT 11 CLOSED

Hi,
The zero-initialized linear layer can be checked in the following code. The zero-initialized linear layer is also used in Big Transfer (Bit) and shares one more related link.

from vit-pytorch.

OK. But where is removing two linear layers and replacing it by a single layer? I only see you zero-initialized a single linear layer.

from vit-pytorch.

Another issue is that you used the Conv2d layer to extract features in the embedding layer. Is this "hybrid architecture" described in the original paper? Could you also support a version which does not rely on CNN?

from vit-pytorch.

As far as I know, the two linear layers in the paper are the MLP layer whose activation function is tanh. You can check that part from the following link (pre_logits and head).
Additionally, in this repository, we create a single linear layer that matches the target class without weight loads of pre_logits and head.

from vit-pytorch.

As far as I know, the hybrid model replaces Conv2d with a different backbone. With a simple implementation you can remove Conv2d and use the BiT backbone for this part. Please refer to timm for detailed implementation of hybrid model.

ViT used Conv2d for patch embedding, and I think not using CNN is a big challenge.

from vit-pytorch.

Please check Equation 1 in the original paper. That's the non-hybrid version.

from vit-pytorch.

The currently implemented part is the same as Equation (1).
Please refer to the following link for the part about Equation (1).

from vit-pytorch.

Line 144 x = self.patch_embeddings(x)

self.patch_embeddings is conv2d?

from vit-pytorch.

That's correct!
Conv2d's kernel_size is the patch size.

from vit-pytorch.

@jeonsworld https://github.com/google-research/vision_transformer/blob/0040316f123353eaba186e7be914e58e656cc120/vit_jax/models.py#L215

Okay. Now I get your point. Look at this comment. The author of ViT said s2d (sequence 2D) + embedding is equal to a Conv operation. I think this is the key. So, in essence, ViT is also CNN-based model. We cannot say we should give up CNN.

from vit-pytorch.

Thanks for the nice comment. I think this gave me a better insight.

from vit-pytorch.