This repository contains code from our paper [Challenging Deep Learning Models with Image Distortion based on the Abutting Grating Illusion] published in Cell Patterns. https://www.cell.com/patterns/fulltext/S2666-3899(23)00026-0

If you use our code or refer to this project, please cite this paper: Jinyu Fan, and Yi Zeng. Challenging Deep Learning Models with Image Distortion based on the Abutting Grating Illusion. Patterns, DOI：https://doi.org/10.1016/j.patter.2023.100695

Even state-of-the-art deep learning models lack fundamental abilities compared with humans. While many image distortions have been proposed to compare deep learning with humans, they depend on mathematical transformations instead of human cognitive functions. Illusory contour has been a popular topic for psychology research, providing insights into visual perception of both humans and various animal species. However, illusory contour perception is seldom studied in deep learning research. The crucial reason is that illusory contour samples are unnatural visual stimuli designed manually, which cannot be directly integrated into the datasets. This paper aims to create illusory contour datasets by distorting existing datasets in deep learning research. To be more specific, we propose an image distortion based on the abutting grating illusion, which is a classic illusory contour phenomenon.

The distortion generates illusory contour perception using line gratings abutting each other. We applied the method to MNIST, high-resolution MNIST, and “16-class-ImageNet” silhouettes. Many models, including models trained from scratch and 109 models pretrained with ImageNet or various data augmentation techniques, were tested. Our results show that abutting grating distortion is challenging even for state-of-the-art deep learning models. We discovered that DeepAugment models outperformed other pretrained models. Visualization of early layers indicates that better-performing models exhibit the endstopping property, which is consistent with neuroscience discoveries. Twenty-four human subjects classified distorted samples to validate the distortion.

The code to generate Abutting Grating Distortion can be found in "utils/abutting_grating_illusions.py"

The following files generate samples of AG distortion AG-MNIST : 1_visualize_mnist28.py high-resolution AG-MNIST : 2_visualize_mnist224.py AG-silhouette ：3_visualize_sil.py

"1_test_mnist28.py" trains models with original MNIST and test them on AG-MNIST. "2_test_mnist224.py" trains models with high-resolution MNIST and test them on high-resolution AG-MNIST.

"1_acc28_visual.py" and "2_acc224_visual.py" plot the accuracy against the epochs during the training.

"3_imagenet_pretrained_test.py" tests pretrained models on the AG-silhouette. Note that the pretrained models we tested are from torchvision and timm packages. We also tested data augmentation models from GitHub, but we do not include them in this repository. You can find the data augmentation models below: https://github.com/hendrycks/imagenet-r https://github.com/clovaai/CutMix-PyTorch https://github.com/bethgelab/game-of-noise https://github.com/google-research/augmix https://github.com/rgeirhos/texture-vs-shape

"3_distribution.py" plots the distribution of models' accuracy. "3_best_performing_models.py" plots the accuracy of best-performing models, which we defined as the outliners in the distributions.

"4_visualize_layer_activation.py" visualizes the average activation of ResNet50 when confronting with the abutting grating samples.

"4_visualize_filter_activation.py" visualizes the globally normalized activation of each filter.

"4_visualising_features.py" visualizes the features of first convolution layer of ResNet50.

In the "human-experiments" folder we provide the code for the human experiments.