Proposing a multimodal hateful memes detection framework

• UNITER outperforms LXMERT and Oscar by a large margin for both their base and large versions

• upsample the confounders in the batches during the supervised fine-tuning step of UNITER
• a loss re-weighting strategy during training and weigh the loss of the hateful class higher, subsequently affecting the update##### s of the model parameters

• AUROC (Area Under the Receiver Operating Characteristics )

• LXMERT, UNITER, Oscar, BERT, R-CNN

• https://arxiv.org/abs/2012.12871
• https://github.com/Nithin-Holla/meme_challenge

AUROC scores of our models on the development set (dev-seen) as well as the Phase 1 and 2 test sets of the challenge.

• Proposed a multimodal hateful memes detection framework that ranked 4th in Phase 2 of the 2020 Hateful Memes Challenge launched by Facebook
• Proposed solution and other state-of-the-art architectures such as Oscar there is a lot of room for improvement in terms of their ability to perform “true” multimodal reasoning.

• the observations while building this model can further inform future research in the field
• Recent developments in the field, such as ERNIE-ViL, a multimodal model that incorporates structured knowledge from scene graphs in its pretraining tasks, is another interesting avenue for future work.

To create an algorithm that identifies multimodal hate speech in internet meme

• Proposed a classification algorithm that combined the multimodal with rules.

Extracted classification rules from training set, and these rules could be used to generate pseudo-label or adjust classification probability. Using rules above, part of samples in test set can almost be determined, expending these samples to training set can greatly improve the metrics of model. adjusted the predictions of each model with rules, then stack models to generate final prediction

Accuracy and AUROC

VisualBERT, VisualBERT COCO, ViLBERT, ViLBERT CC, MMBT-Region, UNITER and Masked Region Modeling(MRM)

• https://arxiv.org/abs/2012.01002

  Scores of top 3 pretrained model

   Scores of top 3 models with MRM

The combination of rules extracted from data set with multimodal framework, which improved both the accuracy and the AUROC more than 13% which leades to accuracy and AUROC of 86.8% and 0.923 respectively.

• Implemented MRM to VisualBert to enhance its effectiveness in Hateful Memes, and applied several common technologies such as K-fold, model stacking, semi-supervised learning, significantly improved the AUROC and accuracy of classification
• The features of the dataset are too conspicuous, this makes it possible for me to extract rules from data, but these rules may not be so practical in real world scenarios

• To focus on improving the multimodal framework