Hi! It was impressive to see your code, especially using parent class (Optimizer) to make SGLD class in sgld_optimizer.py .

However I'm wondering if adding noise with gradient is a little bit wrong. According to the SGLD paper (and also in your ICML Workshop paper), it seems that

langevin_noise = Normal(torch.zeros(size), torch.ones(size) / group['lr'])
p.data.add_(-group['lr'], d_p + langevin_noise.sample().cuda())

based on the fact that lr * (Sample from N(0,1/lr)) is equal to Sample fron N(0,lr)

Hi,
it is not clear to me how the Lr is scaled throughout the learning.

Many Thanks

First of all, thanks for the paper and the source code! Both had influenced the direction of the research topic for my master degree (still in early phase).

KSGLD is mentioned in the paper but I couldn't find the code for KSGLD optimizer anywhere. Did it implemented in some other repository or did I miss something?

Another question (or more of a request for clarification), what does it mean to use SGLD/pSGLD with parameter addnoise=False? I guess:

• SGLD optimizer with addnoise=False is equivalent to running classic SGD optimizer. I tried to verify this by using this SGLD without noise, and compare the result to using PyTorch's SGD optimizer. And the accuracy in each epoch for both optimizer came out almost identical. So I'm almost sure this interpretation is correct, just want to confirm.
• pSGLD optimizer with addnoise=False is equivalent to running classic SGD with RMSProp, or just RMSProp for short, is this correct? (I haven't tried to compare pSGLD without noise to PyTorch's RMSProp optimizer). I have just tried training the model in this repo using PyTorch's RMSprop and the accuracy in each epoch also identical to using pSGLD without noise.