This repo contains the official code of VLM-EMO Context-Aware Emotion Classification in the Wild(VLM-EMO).

In the visual component, we leverage the CLIP image encoder as a foundation and introduce a temporal model, comprised of multiple Transformer Encoders, for capturing temporal features. The ultimate video-level facial features are generated via a trainable class token.

In the textual portion, we employ descriptions that pertain to behavior in lieu of class names as inputs for the text encoder. Additionally, we incorporate a trainable prompt to serve as contextual information for each class's descriptors.