Codes for Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?

CVPR 2019 Paper

If using the code, please cite our paper: BibTex

If you have any question related to the codes or models, please open an issue. If you have general questions about principle of BENN or have any further idea of improving it, please contact us by email: [email protected], [email protected]. Please, no commercial use before getting permission from authors.

Notice: As mentioned in the paper (Section 7) we are aware of the overfitting problem caused by the ensemble technique. If retraining the models, they should basically match the results shown in the paper as well as here, but could be either slightly higher or lower due to random initialization, epoch selection, overfitting, etc. If you have a good idea of how to resolve the overfitting issue of ensemble methods, please contact the authors and we can further improve BENN.

A customized Network-In-Network (NIN) model is used. Please see paper for architecture details.

Generally, we have:

🏠 2 different models (you can specify with --arch allbinnet/nin), corresponding to AB and SB models in the paper

⏳ 2 different training modes (independent training, and sequential training)

⚙️ 5 different ensemble schemes (Bagging, Boost A, Boost B, Boost C, and Boost D)

📊 2 voting strategies (hard majority vote, soft max vote)

For example:

$ python main_bagging_SB.py --epochs 0 --retrain_epochs 100 --root_dir PATH/TO/YOUR/models_bagging_SB/

First download the models from the links above, then run the corresponding python script to test pre-trained models and you should get the exact same numbers comparing with our logs above. For example:

$ python main_bagging_SB.py --epochs 0 --retrain_epochs 0 --root_dir PATH/TO/YOUR/DOWNLOADED/models_bagging_SB/

Notice: For AB models, you should get around 79-82% accuracy for 32 ensembles. For SB models, you should get around 87-89% accuracy for 32 ensembles (usually 15-20 is a reasonable choice due to overfitting). The single BNN should have around 69-73% and 83-84% accuracy for AB and SB model respectively.

The codes and pre-trained models on AlexNet and ResNet-18 will be released soon in near future, please stay tuned. We are currently studying overfitting issue and testing the stability of the gain.

Notice: For AlexNet, you should get around 50-53% (bagging) and 52-55% (boosting) accuracy for 5-6 ensembles. For ResNet-18, you should get around 56-59% (bagging) and 59-62% (boosting) accuracy for 5-6 ensembles. The single BNN should have accuracy around 44% and 48% for AlexNet and ResNet-18. Be sure to use SB model with independent training, and make sure each BNN is well converged before ensemble. Due to overfitting and optimization instability as observed in Section 6.2 from the paper, you may want to train BENN multiple times and pick the best one.

To train BENN for your own application, you can directly reuse the BENN training part of this code. More details will be provided. If you successfully train BENN on some new applications with new architectures and achieve satisfying performance, please contact the authors and we will add a link here.

The single BNN training part of this code is mostly written by referencing XNOR-Net and Jiecao Yu's implementation. Please consider them as well if you use our code. Based on our testing, XNOR-Net is the most stable and reliable open source BNN training scheme with product-level codes.

• Release CIFAR-10 Training Code
• Release CIFAR-10 Pretrained Models
• Release ImageNet Training Code
• Release ImageNet Pretrained Models