This repository is source forge for deep learning study held in KAIST, Mathematical Problem Solving Club in 2019 winter semester.

For those who are not attending this study, but would like to use this repository for your deep learning self study, I separated theoritical parts to lecture notes and practical parts to jupyter notebook practices. I recommend looking both, but it would be enough to see only one.

We will use pytorch as basic tools. Large parts of code will be jupyter notebook.

1. Introduction : What is DL? Lecture1 Practice1
2. MLP : Introduction and Implementation Lecture2 Practice2
3. MLP : Advanced Practice3
4. SGD : Introduction and advanced SGD Lecture3 Practice3
5. Hyperparameter Tuning and How to experiment Lecture4
6. CNN : Introduction and Implementation Lecture5
7. CNN : Advanced
8. RNN : Introduction and Implementation Lecture6
9. RNN : Advanced
10. Further study : Other networks Lecture7
11. Further study : Other methods
12. Individual Presentation

• Difference between AI/ML/DL
• What we will study?
• Problems in DL : Classification, Clustering, Generative Model
• Feed Forward and Backpropagation
• Hidden theory for DL : Universal Approximation Theorem
• How torch works

• What is MLP?
• What is Activation Function?
• What is loss function?
• Simple example using MLP : MNIST
• Batchnorm and Dropout
• Weight Initilization

• Momentum
• Adagrad
• Adadelta
• RMSProp
• Adam
• L-BFGS
• Non Gradient Descent optimizer
  • Genetic Algorithm
• Use of lr scheduler

• Cross Validation
• Lasso/Ridge Regularization
• Hyperparameter Tuning
  • Grid Tuning
  • Random Tuning
  • Bayesian optimization
• Collecting data
  • Preprocessing
  • Benchmarks
  • Data Mining
• Tensorboard

• What is CNN?
• Why CNN > MLP
• Simple example using CNN : MNIST, CIFAR10, CIFAR100
• Reception
• Residual

• What is RNN?
• Why RNN > MLP
• Simple example using RNN : PTB, TIMIT
• LSTM
• GRU
• Attention

• Graph Convolutional Neural Network
• Memory Augmented Neural Network
• Deconvolutional Neural Network
• Modular Neural Network
• How to implement user-defined layer & loss

• Reinforcement Learning
• Semi-supervised Learning
• GAN

• Find some papers from NIPS, ICCV, SIGGRAPH, ICML, IEEE, ICLR, AAAI

Deeplearning StandAlone

Tensorflow Korea Paper Reading

Reinforcement Learning

Bayesian Optimization

Attention