about local reparametrisation trick about bayesian-neural-networks HOT 3 CLOSED

Thanks for your question,

I believe there are no convolutional layers in this repo’s code. I imagine you are referring to variational dropout, as presented in https://papers.nips.cc/paper/5666-variational-dropout-and-the-local-reparameterization-trick.pdf, where activations are multiplied with Gaussian noise. This scenario, can be seen as placing a variational distribution over the weights q(w_{i})=N(w_{i}; \mu_{i}, \alpha\mu_{i}^{2}) where \alpha is shared among all weights. The authors choose to use an improper prior over the weights and compute KL(q(W) || p(W)) using an expansion approximation. If you were to place a Gaussian prior over the weights p(W) = N(W; 0, I), you would calculate the KL term as: KL(N(w_{i}; \mu_{i}, \alpha\mu_{i}^{2}) || p(W)). This expression has a closed-form solution. However, it is unclear to me if this prior is a sensible choice when applying multiplicative Gaussian noise.

In our implementation, we place a factorised Gaussian approximate distribution over weights q(w_{i})=N(w_{i}; \mu_{i},\sigma_{i}^{2}). A Gaussian prior is used. The local reparametrization trick can be applied to these distributions, inducing Gaussians over activations q(a_{j} | x)=N(…). This reduces the variance in the gradient estimator for the likelihood of the data. The KL divergence, however, is still calculated in weight space KL(q(W) || p(W)). This expression can be evaluated in closed form.

from bayesian-neural-networks.

Hi again @ShellingFord221,

I have seen that you have opened multiple issues with theoretical questions. Although the feedback is appreciated, these sections are best left for issues with the actual code. I would refer you to the original work: https://arxiv.org/abs/1505.05424 for a better theoretical understanding of the implemented approaches.

from bayesian-neural-networks.

I'm really sorry that I just want to have some discussions about both implementation and theory of BBB...

from bayesian-neural-networks.