This repository gathers the work on the implementation of the paper Approximate Inference Turns Deep Networks into Gaussian Processes, studied during the course of Bayesian Machine Learning provided by Rémi Bardenet and Julyan Arbel at Ecole Normale Supérieure Paris-Saclay (MVA Master).

You will find our experiments in the experiments.ipynb file and our analysis in the pdf.

Work performed by Pierre Clavier, Emile Cohen, Julia Linhart

This repository contains code to reproduce results for the paper Approximate Inference Turns Deep Networks into Gaussian Processes

The results are readily available in directory /results and can be reproduced by running

python marglik.py --name of_choice

This will produce both toy and real world experiments and save the corresponding measurements to the results directory with a new filename. The plots can then be generated by running

python marglik_plots.py --name original  # use our result files
python marglik_plots.py --name of_choice  # use your result files

We use pre-trained models that are saved in the /models directory and are trained on either CIFAR-10 or MNIST. The plots produced in our paper can be reproduced by running

python kernel_and_predictive.py

To reproduce the plots of kernel, predictive mean, and variance, run

python kernel_and_predictive_plots.py

Use the function dnn2gp.compute_laplace(model, train_loader, prior_prec): to compute a Laplace approximation to the posterior distribution. Simply pass the pytorch model, iterator of samples and targets, and the prior precision (equal to weight-decay parameter). We assume a classification task so we will use the neural network logits (pre-softmax) outputs and assume you trained with softmax.

To obtain Jacobians, GP predictive mean, labels, predictive variance, predictive noise, and predictive mean we can call compute_dnn2gp_quantities(model, data_loader, post_prec=post_prec, limit=1000) with pytorch model that outputs logits and is trained with softmax and the corresponding training data set loader. The posterior precision approximation needs to be passed, too. Otherwise, only the Jacobians, GP predictive mean, and labels are returned.

Note that in correspondence to our paper, we use the reparameterization for these quantities. Only the GP predictive mean corresponds to the dual model in the Theorems while the predictive mean and predictive variances are obtained after reparameterizing for the original data space.

To compute the kernel, the Jacobians from 2. are required. Based on the Jacobians, we can compute the kernel or some aggregation of it with compute_kernel(Jacobians, agg_type='diag'). For most networks and data set sizes, using limit=1000 in will be easily computable and does not take much storage. The aggregation type agg_type denotes how we reduce the NK x NK kernel. 'diag' simply sums up across the diagonals, 'sum' sums all elements of a K x K.

python 3.x with requirements.txt.