Hi,

First I'd like to thank you for this great library of implementation. I find it very helpful in understanding the different variants of GANs.

I think there might be an error in the calculation of the partition function for the Softmax GAN.

The relevant line reads:

Z = tf.reduce_mean(tf.exp(-D_real)) + tf.reduce_mean(tf.exp(-D_fake))

Where I believe if should have been using tf.reduce_sum rather than tf.reduce_mean, as the partition function is a sum, not an average. Perhaps this is accounted for in a further part of the code that I am missing.

Thanks,
o

I tried the vanilla GAN without any modification of your code, but there was always mode collapse happening. What's the possible reason for mode collapse in vanilla GAN?

Thanks

In the code , did you use Wasserstein to count the distance between discriminator and generator ?
I can't see the algorithm of it, could you shed a light ? Thanks

Should be KL with prior and not posterior

D_KL(Q(z|X) || P(z)); calculate in closed form as both dist. are Gaussian

I am reading your least-square GAN code. The G net run twice for training G and D. Why cannot only run G.forward once as the input to G for the two runs are the same?

Hi,

Can you tell me on thing: Are recon_loss and kl_loss in vae tensorflow script scalar or vector?

I tried VAE implementation but did not understand the algo. So I searched for implementations on GitHub and found yours. The problem I am facing with your implementation is to understand 2 things, 1st is what exactly is h_dim and how is the value of it decided?

Thanks in advance

Hi,
I have got an error while running
python infogan_pytorch.py

Extracting ../../MNIST_data/train-images-idx3-ubyte.gz Extracting ../../MNIST_data/train-labels-idx1-ubyte.gz Extracting ../../MNIST_data/t10k-images-idx3-ubyte.gz Extracting ../../MNIST_data/t10k-labels-idx1-ubyte.gz /home/arghyapal/miniconda2/envs/generative-models/lib/python3.5/site-packages/torch/optim/adam.py:74: UserWarning: tensor1 is not broadcastable to self, but they have the same number of elements. Falling back to deprecated pointwise behavior. p.data.addcdiv_(-step_size, exp_avg, denom) Traceback (most recent call last): File "infogan_pytorch.py", line 119, in <module> reset_grad() File "infogan_pytorch.py", line 86, in reset_grad p.grad.data.zero_() AttributeError: 'NoneType' object has no attribute 'data'

Please help me...!!!

I didn't understand where you get the recon_loss term in the AVB implementation from.
I couldn't find it anywhere in the paper.

Amazing repo by the way!

loss = recon_loss + kl_loss

While you should have
loss = recon_loss - kl_loss

To perform gradient descent.

If not you should
loss = -recon_loss + kl_loss
and perform gradient ascent.

The ELbo to maximize is:
ELBO = -DKL + LogLikelihood

Your kl_loss correspond to the -DKL and your binary error to -LogLikelihood. So you have to change your -DKL to be DKL to minimize the ELBO, if not change sign from LogLikelihood and maximize.

Hi, Thank you for the implement of all the deep generative models, and the code is very clean and easy to follow.

However, about the formulate in your post, I think should change X -> Y, and c -> X, or it will cause ambiguity.

I think the D_loss in WGAN(tensorflow) is lack of negative sign

Hi,

Do you know how to reshape image output by your code in order to get inception score (I guess this code is for coloured images).

here is the inception github code:
https://github.com/openai/improved-gan/tree/master/inception_score

Thanks in advance

D_aux = torch.nn.Sequential(
torch.nn.Linear(h_dim, y_dim),
torch.nn.Softmax()
)
should be changed to :
D_aux = torch.nn.Sequential(
torch.nn.Linear(h_dim, y_dim)
)

Because D_aux is the input of nn.cross_entropy(C_real/D_aux, y_true), where D_aux is output of linear() instead of softmax().

I saw that on the official github for that paper
https://github.com/igul222/improved_wgan_training/blob/master/gan_mnist.py#L86

does it make some difference ?

Hi,
If its possible, could you please add our variant of GAN to your collection too? It's a semi-supervised version of infoGAN. Thanks!

https://arxiv.org/abs/1707.04487

seems like you got VAES/ GAN covered

but missing one other according to this article.
https://blog.openai.com/generative-models/

perhaps you can cherry pick this
https://github.com/philkuz/PixelRNN/blob/master/pixelrnn.ipynb

Hello Augustus,

Thank you very much for your code. I am trying to change the Adversarial Autoencoder to use LSTM layers. Have you tried this? Do you think it would work? So far I haven't been able to make it converge.

Thanks!

there are several implementation on the web, I couldn't converge and I'm expecting a small mistake, would be interesting to test your code.

https://github.com/wiseodd/generative-models/blob/master/GAN/infogan/infogan_pytorch.py#L55

how is h = nn.relu(X @ Wxh + bxh.repeat(X.size(0), 1)) valid python? The @ in particular.

Hi,

First of all, thanks so much for these great scripts for for the GANs. They are easy to implement so that I can start off my research. But I have one question in your infogan script. In the code, you use a 10-binary-digit array as your c_noise, the latent code for the generator. There's no problem with it if you only want to use discrete latent code, but what if you want continuous latent code(which was implemented in the original infogan paper)? Or more generally, how do you choose to represent your z_noise and c_noise?

Thanks

Hi,

I think the current implementation for Boundary-seeking GANs is only applicable for continuous variables, and not for discrete which was the main contribution of the paper. Is a discrete implementation in the works?

I've been running the new gibbsnet implementation, but I can't find it to converge with the given settings.

Do you have any examples on what the results should look like?

i think, should use nn.BCEloss, not -(torch.mean(torch.log(D_real)) + torch.mean(torch.log(1 - D_fake)))

WGAN seems really slow, I tried your code with 10 million iterations, but the output images still looks very bad.

Do you have any idea?

Thanks!

Your code about trainig process is very different from the details in the paper(MD GAN)

MDGAN in the original paper must train in the manifold step and the diffusion step.

And the loss also has the differences.

I'm wrong?

May i use python 3?

crossent_loss = torch.mean(-torch.sum(c * torch.log(Q_c_given_x + 1e-8), dim=1))
ent_loss = torch.mean(-torch.sum(c * torch.log(c + 1e-8), dim=1))
mi_loss = crossent_loss + ent_loss

it should be "crossent_loss - ent_loss" I suppose.
the entropy should be -plog(p) (I suppose???)

if it's discontinuous, the entropy should always be zero.
but if it's continuous. it's different....

Hi
Thanks for your wonderful code! I was looking through the code for Bidirectional GAN and have a few doubts :
[1] In the discriminator step why is Line 83 required ? Is it required to update the generator during discriminator learning ? should not it be fixed ?
Is this a G_solver.step() needed ?

[2] while updating the generator(Q and P), are you using the the inverse objective function found in Appendix 2 page 16 here at https://arxiv.org/pdf/1605.09782.pdf ?
otherwise should not it be a minimization problem and the - sign in Line 93 not required ?

[3] also why is the final layer in the generator a sigmoid layer ?

Thanks in advance ! I will be grateful to hear anything on this matter.

Cold you please provide implementation of GAN models in Keras besides tensorflow and pytorch?

I wrote an wgan with gradient penalty on the cifar dataset.
https://github.com/robotcator/wgan-gp/tree/wgan-cifar10
it would be useful to integrate wgan-gp to this repo.

The reference:
pytorch version
original tensorflow version

Hi, I just noticed that the KL Loss in the VAE paper would look like this:
0.5 * torch.sum(torch.exp(logVar) + mean ** 2 - 1. - logVar)

And here, the KL Loss is:
torch.mean(0.5 * torch.sum(torch.exp(logVar) + mean ** 2 - 1. - logVar, 1))

What's your thought in this?

please help me. i am unable to understand how to feed values of x_mb, y_mb for my own dataset instead of mnist.

So I am trying out vanilla gan on a ssh server, and it is telling me there are no display.

I can simply hack around it by commenting out some line, but it is hacking, and I think it is better to have a option which turn it off.

Hi, @wiseodd ,

Thanks for your cool implementation of these models!

I'm a newbie to GANs and tensorflow.
I've tried vanilla gan by running gan_tensorflow.py.

At the end of training, the content of G becomes all 1-s, ie, all the squares just contain the image of number one, not like the snapshot in the mid-term of the training, all numbers are generated.

Are these results correct? How to explain it? Thanks in advance!

Best regards.

Hi, thanks for a useful library!

I notice in the AVB implementation that the discriminator is only a function of z, rather than also one of x. Also, the real and fake samples are swapped in the discriminator loss. Is this on purpose?

In my implementation of AVB, I do exactly as the paper says in Algorithm 1 but I find it's unstable like a vanilla GAN. I have to slow down both learning rates, or make the discriminator Eq (3.3) train slower than the ELBO bound, Eq (3.9)

https://arxiv.org/pdf/1706.00409.pdf

Hello,
I read your blog about cGAN, it's awesome.
and I want to run the code about it using tensorflow.
But here are some mistake under my Ubuntu 16.04 desktop.

1. According to the document of tensorflow. I modify the code from

 tf.concat(concat_dim=1, values=[z, y])
# to 
tf.concat([z,y],1);
# in line 37 and 58

2. there is also a new API design for sigmoid_cross_entropy_with_logits:

#line 90,91
D_loss_real = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(D_logit_real, tf.ones_like(D_logit_real)))
D_loss_fake = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(D_logit_fake, tf.zeros_like(D_logit_fake)))
#to 
D_loss_real = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=D_logit_real, labels=tf.ones_like(D_logit_real)))
D_loss_fake = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=D_logit_fake, labels=tf.zeros_like(D_logit_fake)))

then the code runs.

Although these API changes would not have the influence the understanding of algorithm.
But I think it will help the people who want to run the code and newly to Tensorflow.

Hello Wiseodd,

The source code of GAN and its variations provided here are very helpful for those who are interested in applying GAN in bioinformatics.

My question is how to extract encoded features from an AAE (e.g. aae_pytorch.py in the VAE folder). After reading the codes in aae_pytorch.py, I guess that the following codes can be used to extracted the encoded features of training set of MNIST data.

X = Variable(torch.from_numpy(mnist.train.images))
z_fake = Q(X)
np.savetxt('encoded_feature_MNIST.txt', z_fake.data.numpy())

Am I correct?

Thanks,

Xiangchun

I have used coupled GAN code for my own customize dataset. Discriminator and generator loss not decrease or increase and output noise only. I am unable to understand what is the real problem.
Can you please help me ?
I have tried to debug it but it generate noise images. please help me!
Same generator and discriminator loss.

Hi,

I am a torch user and want to go toward PyTorch. I am asking the question in reference to conditional vae example written in pytorch. We want errors resulting from KL Loss to backpropagate only through the encoder section. In torch, I specifically do that but here I see you calculating loss (both marginal_liklihood and KLLoss) together and simply doing loss.backward(). How does this ensure that KL loss will backprogate only through the encoder? So does this mean, if I have to add any regularizer, I simply calculate loss and do loss.backward() and errors will back propagate through the desired portion within the network?

Thanks.

Hi wiseodd,
I am a little confused . Why you are returning D_prob and hence D_real although you are not using ? I know logits is enough to calculate the loss but why an extra discriminator output you want.

Thanks

Hi,
Thanks for the code, but I found that the gradient penalty term is different from the paper and the code given by the author
gradient_penalty = tf.reduce_mean((slopes-1.)**2)
https://github.com/igul222/improved_wgan_training/blob/fa66c574a54c4916d27c55441d33753dcc78f6bc/gan_mnist.py#L147

However, your code still works well and produce good results. I don't know if you intentionally did it, or it's just a typo but fortunately make no mistakes?

Thanks

Thanks for making these generative models available, they are very interesting.

I have a one hot dataset that I would like to test with these various models, and I am curious what your recommendation would be for converting the existing MNIST dataset import to a dataset of one hot encoded values? I have a flat file that contains 30K lines, each line is one hot encoded with a total of 64 labels, resulting in 5,632 input values/neurons per line.

I was under the mistaken impression that it would be a seamless transition based on the one_hot=True import directive from TensorFlow, only to figure out that the MNIST dataset is float32.

Will I have to use a different loss function with your code to support simple integer valued one hot data?

Thanks in advance

I have run dualgan, but it does not work, just generated noise images.

I checked the code carefully with the paper, it matched, but I am still confused about the G_loss.

I have debugged for a long time, but it's still not work.

Could you please give me some help, and check the code?

Why not add Convolutional Autoencoders?

Hello,
Thanks for your work.

Do you have a plan to integrate with other datasets? MNIST is the easiest one to use but there are many other meaningful datasets such as CelebA. It would be nice if you can provide some codes to test other datasets.