Why resize token embedings when dataset is pororo or flintstone? about arldm HOT 5 CLOSED

Hi, thanks for your interests. I not quite clear about what dose "resized token embeding" means. Could you please refer the corresponding code using a link?

from arldm.

oh, sorry.
I mean why 'clip_embedding_tokens' or 'blip_embedding_tokens' is different when dataset change.

https://github.com/Flash-321/ARLDM/blob/94725e13e7c790ec1025fd5d485771becc367f02/config.yaml#L32-L56
https://github.com/Flash-321/ARLDM/blob/94725e13e7c790ec1025fd5d485771becc367f02/main.py#L119-L121

from arldm.

Thanks for your comments! this is because we added some new tokens for characters in these two datasets.
https://github.com/Flash-321/ARLDM/blob/34b30703a2caeeb2364bdfb161345027217785c6/config.yaml#L35
https://github.com/Flash-321/ARLDM/blob/34b30703a2caeeb2364bdfb161345027217785c6/config.yaml#L42
https://github.com/Flash-321/ARLDM/blob/34b30703a2caeeb2364bdfb161345027217785c6/datasets/flintstones.py#L36-L39
https://github.com/Flash-321/ARLDM/blob/34b30703a2caeeb2364bdfb161345027217785c6/datasets/pororo.py#L37-L40
as a result, the vocab size of tokenizer has been changed, and we need to resize the token embeddings to make sure the embedding layer can still encode the sentence. And the num can be obtained by printing len(clip_tokenizer) and len(blip_tokenizer) after adding those new tokens.

from arldm.

Got it! Thanks!
That is a fantastic work! Looking forward to more work from you.

from arldm.

Thanks! Feel free to open an issue if you have any further questions!

from arldm.