Awesome NAS

A curated list of neural architecture search and related resources. Inspired by awesome-deep-vision, awesome-adversarial-machine-learning, awesome-deep-learning-papers, and awesome-architecture-search.

Please feel free to pull requests or open an issue to add papers.

Table of Contents

• Neural Architecture Search
  • Reinforcement Learning
  • Evolutionary Algorithm
  • Gradient-based
  • Performance Prediction-based
  • Multi-Objective NAS
  • Others
• NAS for Application
• Survey
• Benchmark on ImageNet

NAS

Reinforcement Learning

• Neural Architecture Search with Reinforcement Learning [pdf]
  • Barret Zoph and Quoc V. Le. ICLR 2017
• Designing Neural Network Architectures using Reinforcement Learning [pdf]
  • Baker, Bowen and Gupta, Otkrist and Naik, Nikhil and Raskar, Ramesh. ICLR 2017
• Neural Optimizer Search with Reinforcement Learning [pdf]
  • Bello, Irwan and Zoph, Barret and Vasudevan, Vijay and Le, Quoc V. ICML 2017
• Efficient Architecture Search by Network Transformation [pdf]
  • Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, Jun Wang. AAAI 2018
• Learning Transferable Architectures for Scalable Image Recognition [pdf]
  • Barret Zoph, Vijay Vasudevan, Jonathon Shlens, Quoc V. Le. CVPR 2018
• Practical Block-wise Neural Network Architecture Generation [pdf]
  • Zhong, Zhao and Yan, Junjie and Wu, Wei and Shao, Jing and Liu, Cheng-Lin. CVPR 2018
• Efficient Neural Architecture Search via Parameter Sharing [pdf]
  • Pham, Hieu and Guan, Melody Y and Zoph, Barret and Le, Quoc V and Dean, Jeff. ICML 2018
• MnasNet: Platform-Aware Neural Architecture Search for Mobile [pdf] [code]
  • Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Quoc V. Le. arXiv 2018.07
• Path-Level Network Transformation for Efficient Architecture Search
  • Cai, Han and Yang, Jiacheng and Zhang, Weinan and Han, Song and Yu, Yong. ICML 2018
• ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
  • Cai, Han and Zhu, Ligeng and Han, Song. ICLR 2019

Evolutionary Algorithm

• Population Based Training of Neural Networks [pdf]
  • Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, Chrisantha Fernando, Koray Kavukcuoglu. arXiv 1711
• Large-Scale Evolution of Image Classifiers [pdf]
  • Real, Esteban and Moore, Sherry and Selle, Andrew and Saxena, Saurabh and Suematsu, Yutaka Leon and Tan, Jie and Le, Quoc and Kurakin, Alex. ICML 2017
• Hierarchical Representations for Efficient Architecture Search [pdf]
  • Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, Koray Kavukcuoglu. ICLR 2018
• Regularized Evolution for Image Classifier Architecture Search [pdf]
  • Real, Esteban and Aggarwal, Alok and Huang, Yanping and Le, Quoc V. ICML 2018 Workshop

Gradient-based Approach

• Differentiable Neural Network Architecture Search [pdf]
  • Richard Shin, Charles Packer, Dawn Song, ICLR 2018 Workshop
• Understanding and Simplifying One-Shot Architecture Search [pdf]
  • Bender, Gabriel and Kindermans, Pieter-Jan and Zoph, Barret and Vasudevan, Vijay and Le, Quoc. ICML 2018
• SMASH: One-Shot Model Architecture Search through HyperNetworks [pdf]
  • Brock, Andrew and Lim, Theodore and Ritchie, James M and Weston, Nick. ICLR 2018
• Neural Architecture Optimization [pdf] [code]
  • Luo, Renqian and Tian, Fei and Qin, Tao and Liu, Tie-Yan. NIPS 2018
• DARTS: Differentiable Architecture Search [pdf] [code]
  • Hanxiao Liu, Karen Simonyan, Yiming Yang. ICLR 2019
• Graph HyperNetworks for Neural Architecture Search [pdf]
  • Chris Zhang, Mengye Ren, Raquel Urtasun. ICLR 2019
• SNAS: stochastic neural architecture search
  • Sirui Xie, Hehui Zheng, Chunxiao Liu, Liang Lin. ICLR 2019

Performance Prediction-based

• Speeding up Automatic Hyperparameter Optimization of Deep Neural Networksby Extrapolation of Learning Curves [pdf] [code]
  • Tobias Domhan, Jost Tobias Springenberg, Frank Hutter. IJCAI 2015
• Learning Curve Prediction with Bayesian Neural Networks [pdf]
  • Aaron Klein, Stefan Falkner, Jost Tobias Springenberg, Frank Hutter. ICLR 2017
• Progressive Neural Architecture Search [pdf] [code]
  • Chenxi Liu, Barret Zoph, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, Kevin Murphy. ECCV 2018
• Accelerating Neural Architecture Search using Performance Prediction [pdf]
  • Bowen Baker, Otkrist Gupta, Ramesh Raskar, Nikhil Naik. ICLR 2018 Workshop

Multi-Objective NAS

• NSGA-NET: A Multi-Objective Genetic Algorithm for Neural Architecture Search [pdf]
  • Lu, Zhichao and Whalen, Ian and Boddeti, Vishnu and Dhebar, Yashesh and Deb, Kalyanmoy and Goodman, Erik and Banzhaf, Wolfgang, arXiv 1810
• Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution
  • Thomas Elsken, Jan Hendrik Metzen, Frank Hutter. ICLR 2019

Others

• Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization [pdf]
  • Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, Ameet Talwalkar. ICLR 2017
• Hyperparameter Optimization: A Spectral Approach [pdf] [code]
  • Elad Hazan, Adam Klivans, Yang Yuan. NIPS 2017 Workshop
• Neural Architecture Search with Bayesian Optimisation and Optimal Transport [pdf]
  • Kandasamy, Kirthevasan and Neiswanger, Willie and Schneider, Jeff and Poczos, Barnabas and Xing, Eric. NIPS 2018

NAS for Application

• Fast Neural Architecture Search of Compact Semantic Segmentation Models via Auxiliary Cells [pdf]
  • Nekrasov, Vladimir and Chen, Hao and Shen, Chunhua and Reid, Ian. arXiv 1810
• Training Frankenstein’s Creature to Stack: HyperTree Architecture Search [pdf]
  • Andrew Hundt, Varun Jain, Chris Paxton, Gregory D. Hager. arXiv 1810
• Searching for efficient multi-scale architectures for dense image prediction [pdf]
  • Chen, Liang-Chieh and Collins, Maxwell D and Zhu, Yukun and Papandreou, George and Zoph, Barret and Schroff, Florian and Adam, Hartwig and Shlens, Jonathon. NIPS 2018
• Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation [pdf]
  • arXiv 1901

Survey

• Neural Architecture Search: A Survey
  • Thomas Elsken, Jan Hendrik Metzen, Frank Hutter. arXiv 1808
• Taking Human out of Learning Applications: A Survey on Automated Machine Learning [pdf]
  • Yao Quanming, Wang Mengshuo, Jair Escalante Hugo, Guyon Isabelle, Hu Yi-Qi, Li Yu-Feng, Tu Wei-Wei, Yang Qiang, Yu Yang. arXiv 1810

Benchmark on ImageNet