Code to reproduce the NeurIPS 2022 Paper : "Do Residual Neural Networks discretize Neural Ordinary Differential Equations?"
This package has been developed and tested with python3.8
. It is therefore not guaranteed to work with earlier versions of python.
This package can easily be installed using pip
, with the following command:
pip install -e .
This will install the package and all its dependencies, listed in requirements.txt
. To test that the installation has been successful, you can install pytest
and run the test suite using
pip install pytest
pytest
python experiments/expe_linear_weights.py
The plot is saved in the folder figures.
On CIFAR:
python experiments/one_expe_CIFAR_10.py -m 'iresnetauto' --n_layers 8
the argument --n_layers correspond to the number of residual block per layer.
On ImageNet (needs cuda):
python experiments/one_expe_image_net.py -m 'iresnetauto' --n_layers 8
Experiment in Figure 3 - (b) Failure of the adjoint method with a ResNet-101 on ImageNet (needs cuda)
python experiments/one_expe_image_net.py --use_backprop True
python experiments/one_expe_image_net.py --use_backprop False
For CIFAR (pretrained model is available):
python experiments/one_expe_finetuning_CIFAR.py
For ImageNet (needs cuda and model needs to be pretrained first):
python experiments/one_expe_finetuning_imagenet.py
For experiments related to Figures 3 (c) and 4, see next section.
Go to the resnet_cifar folder:
cd resnet_cifar
python one_expe_cifar.py --backprop 1 --heun 0 --depth 16
Available arguments are 1 (True) or 0 (False) for --backprop and --heun. Available arguments for --depth can be any integer. Make sure to run on all the possible combinations for --backprop and --heun in order to reproduce the experiment.
python get_results.py
(Optionally modify the array depths in get_results.py to cover the different values for depth)
Set the value of n_tries in plot_paper_heun.py to the amount of seeds you used. Then run
python plot_paper_heun.py
Compare the gradients when using our Adjoint Method or not:
python compare_gradients.py --depth 32
The optional argument --heun can be passed in order to compare gradients for Heun. Results are saved automatically in metrics_gradient.
Plot the results
python plot_grad_dist.py
To confirm that the memory requirements for the adjoint method are much smaller than for standard backpropagation, you can run from the root folder:
python experiments/memory.py
(This can take a few seconds.)
Results are plotted in the folder figures.
If you use this code in your project, please cite::
Michael E. Sander, Pierre Ablin, Gabriel Peyré
Do Residual Neural Networks discretize Neural Ordinary Differential Equations?
NeurIPS 2022
https://arxiv.org/abs/2205.14612