Thesis project for the B.Sc. in Computer Science at Goethe University

The models folder holds the implementation of the different computational models that were used in this study. All RDMs and Analysis was put into the folders of the respective dataset.

The jupyter notebooks displaying RDMs of the Algonauts dataset are too large to display in browser and need to be downloaded in order to be viewable, however all other code is visible.

Ever since the inception of Artificial Neural Networks (ANNs) and especially since they have begun performing substantially well on various tasks, researchers have been interested in studying how the representations learned by the models relate to the human brain. The majority of research efforts have focused on unimodal models, which possess limitations when compared to the human brain due to their reliance on a single modality. However, recent advances in deep learning have led to the development of multimodal models, capable of processing and integrating information from multiple modalities, more akin to the human brain. These multimodal models hold the promise of bridging the gap between artificial intelligence and human cognitive processes, leading to a more profound understanding of information processing. This study explores the similarity of architecturally diverse vision-and-language models (CLIP, ALBEF, ViLT) to the human brain in comparison to unimodal models (GPT-2, BERT, ViT). Results, though exploratory in nature, show that multimodal models across both the visual and language domain are higher correlated with the human cognitive system than the unimodal models they are based on.