Course Material: Numerical Methods for Deep Learning

This repository contains lecture slides and MATLAB codes for a one-semester graduate-level class on numerical methods for deep learning.

This course was first designed and taught by Lars Ruthotto at Emory University and Eldad Haber at University of British Columbia in the Spring of 2018.

Scope

This course provides students with the mathematical background needed to analyze and further develop numerical methods at the heart of deep learning. The course is simultaneously geared towards science students, who may be interested/experienced with deep learning and would like to strengthen their theoretical foundation, and mathematics students, who may have a background in numerical analysis and who are curious about the emerging technology of deep learning. The course primarily targets graduate students. However, parts of the material may be accessible to advanced undergraduate studens.

The ultimate goal is to implement a simple, mathematically sound deep learning code from scratch. This look under the hood will provide valuable insight into successful and well-tested software packages for deep learning tasks and understand their success (and failure). Students will use MATLAB to code basic concepts.

Course Outline

1. Introduction and Overview

Module 1: Linear Regression and Classification

1. Linear Models and Least-Squares
2. Iterative Methods for Least-Squares
3. Linear Models for Classification
4. Newton's Method for Classification
5. Regularization for Image Classification

Module 2: Neural Networks

1. Introduction to Nonlinear Models
2. Single Layer Neural Networks
3. Training Algorithms for Single Layer Neural Networks
4. Introduction to Deep Neural Networks
5. Differentiating Deep Neural Networks
6. Stochastic Gradient Descent and Variants

Module 3: Parametric Models and Convolutional Neural Networks

1. Introduction to Parametric Models
2. Application of CNN: Image Segmentation
3. CNN and their relation to PDEs

Acknowledgements

Development of this material is in part supported by the National Science Foundation under Grant Numbers 1522599 and 1751636. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.