ucla-vision / entropy-sgd Goto Github PK

In python/optim.py, at line 71, you do this:
dw = w.grad.data
Is it implement the equation of 4:
The paper's Algorith 1.
If so, does dw = w.grad.data do the forward and backward of the network, and, how does it work? Can you explain this clearly?
Thanks!

In the paper (line 5 of Algorithm 1), the dummy weights are updated as

x' <- x' - \eta' dx' + \sqrt{\eta'}\epsilon N(0,1)

But in both the lua and pytorch implementations instead of multiplying by the square root of the inner learning rate the update looks like this:

Lua:
dx:add(-g, xc-lx):add(wd,lx):add(noise/math.sqrt(0.5*lclr), eta)

Pytorch:
dw.add_(-g, wc-w.data).add_(eps/np.sqrt(0.5*llr), eta)

Is there a contradiction here?

model and criterion arguments at step method (https://github.com/ucla-vision/entropy-sgd/blob/master/python/optim.py#L19) are not used at all in the Python code. Are these really necessary?

Hi, i have submit an issue at Pytorch with entropy-sgd. Could you help us to find the reason of the much slower process? Thank you very much!

Thanks for very nice algorithm and implementation. I have a question about the noise term, which reads as follows in the implementation:

ldfdx:add(-g, xc-lx):add(wd,lx):add(noise/math.sqrt(0.5*lclr), eta)

but according to line 5 of Algorithm 1 in paper, shouldn't it be:

ldfdx:add(-g, xc-lx):add(wd,lx):add(noise*math.sqrt(lclr), eta)

Why is noise divided by lclr and lclr is multiplied by 0.5?

Thanks a lot!