pytorch code for Improved Training of Wasserstein GANs about wassersteingan HOT 18 CLOSED

Might be possible to implement improved training in PyTorch now, the merge was done yesterday 👍

from wassersteingan.

@NickShahML you can subscribe to the PR

from wassersteingan.

Hi Anton,

This cannot be implemented in pytorch until we merge the PR pytorch/pytorch#1016 which enables gradients of gradients

from wassersteingan.

@pclucas14 Please follow this PR for info on double backward for convolutions: pytorch/pytorch#1643

from wassersteingan.

I think on your input variable, you can just set requires_grad=True, and then take torch.norm(your_input.grad, 2)

from wassersteingan.

https://github.com/igul222/improved_wgan_training

edit: The early morning getting to me, you wanted pytorch 😛

from wassersteingan.

@atgambardella One needs the gradient of the gradient norm w.r.t. parameters, not the gradient w.r.t. the input

@ccurro This is a tensorflow code. I'm not aware of any pytorch implementation

from wassersteingan.

@aosokin Section 4 of the paper, "Sampling along straight lines says":

The gradient term ||∇xˆD(xˆ)||2 is with respect to the points xˆ, not the parameters of D

from wassersteingan.

@atgambardella agreed, my bad.
However this does not solve the problem. One needs to differentiate this norm, not simply compute it.

from wassersteingan.

@soumith Thanks a lot for your reply! Looking forward to see that happen!

from wassersteingan.

Is there a way to know when this merger has occurred?

from wassersteingan.

Is it possible to implement it in Torch? How can we compute the gradient of the norm of the gradient of D?

from wassersteingan.

This repository has an implementation of it using the new gradients of gradients feature: https://github.com/caogang/wgan-gp.

from wassersteingan.

@jfsantos gradients of gradients? what do you mean by that?

from wassersteingan.

@rafaelvalle For normalizing w.r.t. the norm of the gradients of the discriminator, you need to be able to backprop through that gradient, so you need to compute the gradient of a gradient. This is a recent feature in PyTorch's autograd.

from wassersteingan.

@jfsantos yes! I got confused. w.r.t. the norm of the gradients of the discriminator on the interpolated points between Pr and Pg!

from wassersteingan.

@rafaelvalle @jfsantos but convolutions are not supported yet I believe.

from wassersteingan.

Hi,
does anyone know when convolutions will be supported ?

from wassersteingan.