Benchmarks for every pre-trained model in PyTorch and Keras-Tensorflow. Benchmarks are reproducible.

Combining Keras and PyTorch benchmarks into a single framework lets researchers decide which platform is best for a given model. For example resnet architectures perform better in PyTorch and inception architectures perform better in Keras (see below). These benchmarks serve as a standard from which to start new projects or debug current implementations.

For researchers exploring Keras and PyTorch models, these benchmarks serve as a standard from which to start new projects or debug current implementations.

Many researchers struggle with reproducible accuracy benchmarks of pre-trained Keras (Tensorflow) models on ImageNet. Examples of issues are here1, here2, here3, here4, and here5.

In Keras, the published benchmarks on Keras Applications cannot be reproduced by exactly copying the associated code. In fact, the reported accuracies are usually higher than the actual accuries.

I dive slightly deeper into the reproducibility issues of Keras in the associated blog post.

The actual validation set accuracies on ImageNet for all Keras and PyTorch models (verified on macOS 10.11.6, Linux Debian 9, and Ubuntu 18.04).

• Download

  • Download the Imagenet 2012 Validation dataset from http://image-net.org/download-images or other options. This dataset contains 50000 images

• Preprocess/Extract validation data

  • Once ILSVRC2012_img_val.tar is downloaded, run:

  # Credit to Soumith: https://github.com/soumith/imagenet-multiGPU.torch
  $ cd ../ && mkdir val && mv ILSVRC2012_img_val.tar val/ && cd val && tar -xvf ILSVRC2012_img_val.tar
  $ wget -qO- https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh | bash

The top 5 predictions for every example in the ImageNet validation set have been pre-computed for you here for Keras models and here for PyTorch models. These are automatically used by the following code which takes a few seconds to run:

$ git clone https://github.com:cgnorthcutt/imagenet-benchmarking.git
$ cd benchmarking-keras-pytorch
$ python imagenet_benchmarking.py /path/to/imagenet_val_data

You can also reproduce the inference-time output of each Keras and PyTorch model without using the pre-computed data. Inference for Keras takes a long time (5-10 hours) because I compute the forward pass on each example one at a time and avoid vectorized operations: this was the only approach I found would reliably reproduce the same accuracies. PyTorch is fairly quick (less than one hour). To reproduce:

$ git clone https://github.com:cgnorthcutt/imagenet-benchmarking.git
$ cd benchmarking-keras-pytorch
$ # Compute outputs of PyTorch models (1 hour)
$ ./imagenet_pytorch_get_predictions.py /path/to/imagenet_val_data
$ # Compute outputs of Keras models (5-10 hours)
$ ./imagenet_keras_get_predictions.py /path/to/imagenet_val_data
$ # View benchmark results
$ ./imagenet_benchmarking.py /path/to/imagenet_val_data

You can control GPU usage, batch size, output storage directories, and more. Run the files with the -h flag to see command line argument options.

One of the goals of this project is to help reconcile issues with reproducibility in Keras pre-trained models. The way I deal with these issues is three-fold. In Keras I

1. avoid batches during inference.
2. run each example one at a time. This is silly slow, but yields a reproducible output for every model.
3. only run models in local functions or use with clauses to ensure no aspects of a previous model persist in memory when the next model is loaded.

Copyright (c) 2019 Curtis Northcutt. Released under the MIT License. See LICENSE for details.