Giter VIP home page Giter VIP logo

bbi's Introduction

Born-Infeld (BI) for AI: Energy-Conserving Descent (ECD) for Optimization

This repository contains the code for the BBI optimizer, introduced in the paper Born-Infeld (BI) for AI: Energy-Conserving Descent (ECD) for Optimization. 2201.11137. It is implemented using Pytorch.

The repository also includes the code needed to reproduce all the experiments presented in the paper. In particular:

  • The BBI optimizer is implemented in the file inflation.py.

  • The jupyter notebooks with the synthetic experiments are in the folder synthetic. All the notebooks already include the output, and text files with results are also included in the folder. In particular

    • The notebook ackley.ipynb can be used to reproduce the results in Sec. 4.1.
    • The notebook zakharov.ipynb can be used to reproduce the results in Sec. 4.2.
    • The notebook multi_basin.ipynb can be used to reproduce the results in Sec. 4.3.
  • The ML benchmarks described in Sec. 4.5 can be found in the folders CIFAR and MNIST. The notebooks already include some results that can be inspected, but not all the statistics that builds up the results in Table 2. In particular:

    • CIFAR : The notebook CIFAR-notebook.ipynb uses hyperopt to estimate the best hyperparameters for each optimizer and then runs a long run with the best estimated hyperparamers. The results can be analyzed with the notebook analysis-cifar.ipynb, which can also be used to generate more runs with the best hyperparameters to gather more statistics. The subfolder results already includes some runs that can be inspected.

    • MNIST: The notebooks mnist_scan_BBI.ipynb and mnist_scan_SGD.ipynb perform a grid scan using BBI and SGD, respectively and gather some small statistics. All the results are within the notebooks themselves.

  • The PDE experiments can be run by running the script script-PDE.sh as

    bash script-PDE.sh
    

    This will solve the PDE outlined in Sec. 4.4 and App. C multiple times with the same initialization. The hyperparameters are also kept fixed and can be obtained from the script itself. In particular:

    • feature 1 means that an L2 regularization is added to the loss.
    • seed specifies the seed, which fixes the initialization of the network. The difference between the different runs then is only due to the random bounces, which are not affected by this choice of the seed.

    The folder results already includes some runs. The runs performed in this way are not noisy, i.e. the set of points sampled from the domain is kept fixed. To randomly change the points every "epoch" (1000 iterations), edit the file experiments/PDE_PoissonD.py by changing line 134 to self.update_points = True.

The code has been tested with Python 3.9, Pytorch 1.10, hyperopt 0.2.5. We ran the synthetic experiments and MNIST on a six-core i7-9850H CPU with 16 GB of RAM, while we ran the CIFAR and PDE experiments on a pair of GPUs. We tested both on a pair of NVIDIA GeForce RTX 2080 Ti and on a pair of NVIDIA Tesla V100-SXM2-16GB GPUs, coupled with 32 GB of RAM and AMD EPYC 7502P CPUs.

The Resnet-18 code (in experiments/models) and the utils.py helper functions are adapted from https://github.com/kuangliu/pytorch-cifar (MIT License).

bbi's People

Contributors

gbdl avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.