Code to reproduce the NeurIPS 2022 Paper : "Do Residual Neural Networks discretize Neural Ordinary Differential Equations?"

Compat

This package has been developed and tested with python3.8. It is therefore not guaranteed to work with earlier versions of python.

Install the repository on your machine

This package can easily be installed using pip, with the following command:

pip install -e .

This will install the package and all its dependencies, listed in requirements.txt. To test that the installation has been successful, you can install pytest and run the test suite using

pip install pytest
pytest

Reproducing the experiments/figures of the paper

Experiment in Figure 2 - Illustration of the smoothness of the weights in the linear case

python experiments/expe_linear_weights.py

The plot is saved in the folder figures.

Experiment in Figure 3 - (a) Train models with tied weights

On CIFAR:

python experiments/one_expe_CIFAR_10.py -m 'iresnetauto' --n_layers 8

the argument --n_layers correspond to the number of residual block per layer.

On ImageNet (needs cuda):

python experiments/one_expe_image_net.py -m 'iresnetauto' --n_layers 8

Experiment in Figure 3 - (b) Failure of the adjoint method with a ResNet-101 on ImageNet (needs cuda)

python experiments/one_expe_image_net.py --use_backprop True
python experiments/one_expe_image_net.py --use_backprop False

Experiment in Table 2 - Refine pretrained models by untying their weights

For CIFAR (pretrained model is available):

python experiments/one_expe_finetuning_CIFAR.py

For ImageNet (needs cuda and model needs to be pretrained first):

python experiments/one_expe_finetuning_imagenet.py

For experiments related to Figures 3 (c) and 4, see next section.

Train our simple ResNet on CIFAR with Euler or Heun / with Adjoint Method or Backpropagation

Go to the resnet_cifar folder:

cd resnet_cifar

Training

python one_expe_cifar.py --backprop 1 --heun 0 --depth 16

Available arguments are 1 (True) or 0 (False) for --backprop and --heun. Available arguments for --depth can be any integer. Make sure to run on all the possible combinations for --backprop and --heun in order to reproduce the experiment.

Getting the results

python get_results.py

(Optionally modify the array depths in get_results.py to cover the different values for depth)

Plot

Set the value of n_tries in plot_paper_heun.py to the amount of seeds you used. Then run

python plot_paper_heun.py

Gradients

Compare the gradients when using our Adjoint Method or not:

python compare_gradients.py --depth 32

The optional argument --heun can be passed in order to compare gradients for Heun. Results are saved automatically in metrics_gradient.

Plot the relative norms of the gradients:

Plot the results

python plot_grad_dist.py

Memory savings:

To confirm that the memory requirements for the adjoint method are much smaller than for standard backpropagation, you can run from the root folder:

python experiments/memory.py

(This can take a few seconds.)

Results are plotted in the folder figures.

Cite

If you use this code in your project, please cite::

Michael E. Sander, Pierre Ablin, Gabriel Peyré
Do Residual Neural Networks discretize Neural Ordinary Differential Equations?
NeurIPS 2022
https://arxiv.org/abs/2205.14612

Issue with setuptools and versioning

I get the following error when trying to do a setup for the library. It seems that neither numpy distutils, nor setuptools are working:

Using numpy distutils:

Obtaining file:///E:/Alt/SideProjects/resnet_nodes
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [72 lines of output]
running egg_info
E:\Alt\SideProjects\resnet_nodes\setup.py:5: DeprecationWarning:

    `numpy.distutils` is deprecated since NumPy 1.23.0, as a result
    of the deprecation of `distutils` itself. It will be removed for
    Python >= 3.12. For older Python versions it will remain present.
    It is recommended to use `setuptools < 60.0` for those Python versions.
    For more details, see:
      https://numpy.org/devdocs/reference/distutils_status_migration.html


    from numpy.distutils.core import setup
  D:\SoftwareFiles\Anaconda\envs\resnet_nodes\lib\site-packages\setuptools\dist.py:520: SetuptoolsDeprecationWarning: Invalid version: '"0.0.dev"'.
  !!

          ********************************************************************************
          The version specified is not a valid version according to PEP 440.
          This may not work as expected with newer versions of
          setuptools, pip, and PyPI.

          By 2023-Sep-26, you need to update your project and remove deprecated calls
          or your builds will no longer be supported.

          See https://peps.python.org/pep-0440/ for details.
          ********************************************************************************

  !!
    self._validate_version(self.metadata.version)
  D:\SoftwareFiles\Anaconda\envs\resnet_nodes\lib\site-packages\setuptools\command\egg_info.py:131: SetuptoolsDeprecationWarning: Invalid version: '"0.0.dev"'.
  !!

          ********************************************************************************
          Version '"0.0.dev"' is not valid according to PEP 440.

          Please make sure to specify a valid version for your package.
  setuptools.extern.packaging.version.InvalidVersion: Invalid version: '-0.0.dev-'
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

When using setuptools

Obtaining file:///E:/Alt/SideProjects/resnet_nodes
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [59 lines of output]
running egg_info
D:\SoftwareFiles\Anaconda\envs\resnet_nodes\lib\site-packages\setuptools\dist.py:520: SetuptoolsDeprecationWarning: Invalid version: '"0.0.dev"'.
!!
          ********************************************************************************
          The version specified is not a valid version according to PEP 440.
          This may not work as expected with newer versions of
          setuptools, pip, and PyPI.

          By 2023-Sep-26, you need to update your project and remove deprecated calls
          or your builds will no longer be supported.

          See https://peps.python.org/pep-0440/ for details.
          ********************************************************************************

  !!
    self._validate_version(self.metadata.version)
  D:\SoftwareFiles\Anaconda\envs\resnet_nodes\lib\site-packages\setuptools\command\egg_info.py:131: SetuptoolsDeprecationWarning: Invalid version: '"0.0.dev"'.
  !!

          ********************************************************************************
          Version '"0.0.dev"' is not valid according to PEP 440.
  setuptools.extern.packaging.version.InvalidVersion: Invalid version: '-0.0.dev-'
  [end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

michaelsdr / resnet_nodes Goto Github PK

resnet_nodes's Introduction

Code to reproduce the NeurIPS 2022 Paper : "Do Residual Neural Networks discretize Neural Ordinary Differential Equations?"

Compat

Install the repository on your machine

Reproducing the experiments/figures of the paper

Experiment in Figure 2 - Illustration of the smoothness of the weights in the linear case

Experiment in Figure 3 - (a) Train models with tied weights

Experiment in Figure 3 - (b) Failure of the adjoint method with a ResNet-101 on ImageNet (needs cuda)

Experiment in Table 2 - Refine pretrained models by untying their weights

Train our simple ResNet on CIFAR with Euler or Heun / with Adjoint Method or Backpropagation

Training

Getting the results

Plot

Gradients

Plot the relative norms of the gradients:

Memory savings:

Cite

resnet_nodes's People

Contributors

Stargazers

Watchers

resnet_nodes's Issues

Using numpy distutils:

When using setuptools

Recommend Projects

Recommend Topics

Recommend Org