Implementation of a classical architecture from the paper "ImageNet Classification with Deep Convolutional Neural Networks" by Alex Krizhevsky , Ilya Sutskever , Geoffrey E. Hinton. on CIFAR10 dataset.
- Local Response Normalization
- Overlapping Pooling
- Deep Convolutional Network
The architecture of the AlexNet consists of the following layer blocks:
- (1) Conv2D + ReLU + LocalResponseNorm + MaxPooling (x2)
- (2) Conv2D + ReLU + MaxPooling (x1)
- (3) Conv2D + ReLU (x2)
- (4) Dropout + Linear + ReLU (x2)
- (5) Linear (x1)
Due to limited computational resources the significantly smaller dataset than ImageNet had to be chosen. CIFAR10 is a toy, well known dataset small dataset which consists of 60000 32x32x3 coloured images of 10 different classes (airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck).
net.py
- contains model implementation
train.py
- contains train_model
function responsible to taking care of training
main.py
- contains parsing arguments as well as main program loop
The usage is really straightforward. We call python main.py <flags>
where <flags>
indicate chosen flags.
load_model_path
(default=None) - path to load trained weights
save
(default='/checkpoints/model_<current_time>.pth') - path to save the model
train
(default=True) - flag controls whether we want to train the model or just used trained model to predict
resize
(default=[70, 70]) - resize input images before fitting into the model
random_crop
(default=[64, 64]) - random crop input images before fitting into the model
mean
(default=[0.5, 0.5]) - mean for normalization of the input images
std
(default=[0.5, 0.5]) - std for normalization of the input images
batch_size
(default=256) - size of the training batch
val_size
(default=-0.1) - percentage of the validation set
num_epochs
(default=25) - number of epochs
device
(default='cpu') - device on which the training will be run
log_freq
(default=50) - number of iterations between logging current state of the training
logger
(default=50) - number of iterations between logging current state of the training
fraction_of_data
(default=1.0) - percentage of the dataset used for training
Krizhevsky, Alex and Sutskever, Ilya and Hinton, Geoffrey E.
We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overfitting in the fully-connected layers we employed a recently-developed regularization method called “dropout” that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry.
@inproceedings{10.5555/2999134.2999257,
author = {Krizhevsky, Alex and Sutskever, Ilya and Hinton, Geoffrey E.},
title = {ImageNet Classification with Deep Convolutional Neural Networks},
year = {2012},
publisher = {Curran Associates Inc.},
address = {Red Hook, NY, USA},
abstract = {We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overriding in the fully-connected layers we employed a recently-developed regularization method called "dropout" that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry.},
booktitle = {Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1},
pages = {1097–1105},
numpages = {9},
location = {Lake Tahoe, Nevada},
series = {NIPS'12}
}