based on CS231n "Visualizing and Understanding"

It is very nature to think about what have the kernels in the CNN learned during the training process. The visualizing and understanding CNN plays an important role in CNN validating.

Visualize the kernels from the first layer. We can also visualize filters at higher layers, but not that interesting.

The last layer here means the layer immediately before the classifier. So it is actually a feature vector.

(from ImageNet Classification with Deep Convolutional Neural Networks)

The idea is to show the top k-th training image that produce feature vectors with the smallest Euclidean distance from the feature vector for the test image.

(from Visualizing Data using t-SNE)

Visualize the space of the feature vectors b reducing the dimensionality of vectors from high dimensions to 2 dimensions using simple algorithm PCA or more complex algorithm t-SNE.

Visualize each feature map at some certain layer as grayscale images.

(from Striving for Simplicity: The All Convolutional Net)

1. Pick a layer and a channel; e.g. conv5 is $1281313$, pick the channel $17/128$
2. Run many images through the network record values of chosen channel
3. visualize image patches that correspond to maximal activations

Mask part of the image before feeding to CNN, check how much predicted probabilities change.

T.B.C.