kasparmartens / neuralprocesses Goto Github PK

Hi Kaspar,

I tried replicating the results in your post in PyTorch, and I'm unable to get even close to the kind of results you display on your blog. I am sure there is an error on my end somewhere, but I have poured over the paper and your code and your blog post and I'm unable to see anything that could be behind it. I had a friend look over my implementation too and were unable to spot anything of substance different.

I have tried as best as possible to follow the architecture and setup of your experiment 1, and I see very different behavior. The code is very simple, It's afterall only a handful of simple nns,

https://github.com/chrisorm/Machine-Learning/blob/ngp/Neural%20GP.ipynb

Some things I witness that you don't seem to see (shown in the notebook):
-My q distribution concentrates (i.e. std goes to 0).

• Related to above, I see no substantial difference between function samples when extrapolating outside of the data like you seem to.
  -My prior function samples display substantially less variance than yours seem to.

The first led me to suspect an error in my KLD term, but that does not seem to be the case - I unit tested my implementation and I think it is correct. The loss looks good and the network clearly converges.

The second is a bit stranger - do you perhaps use some particular initialization of the weights to draw these samples, over and above setting z ~ N(0,1)?

Would you happen to have any insights as to what may be behind this difference?

Thanks for taking the time to do your post, it has some really great insights into the method!

Chris

I believe there is a mistake in the way that you calculate the KL-divergence. The log term should be subtracted instead of added.

KLqp_gaussian <- function(mu_q, sigma_q, mu_p, sigma_p){
  sigma2_q <- tf$square(sigma_q) + 1e-16
  sigma2_p <- tf$square(sigma_p) + 1e-16
  temp <- sigma2_q / sigma2_p + tf$square(mu_q - mu_p) / sigma2_p - 1.0 - tf$log(sigma2_p / sigma2_q + 1e-16)
  0.5 * tf$reduce_sum(temp)
}

I think your coding about NP exists several problems. I suggest you should check your code again and again.

If we wanted to make this bit

Hi Kaspar,
Thank you so much for sharing your code! I have implemented a tensorflow version and it works well on sin functions. However, when it comes to the functions sampled from gaussian processes, the prediction cannot capture the complexity of the functions. Have you had similar problem?
Thank you!
Yangg-S

Did you try to do the MNIST image completion task with your neural process implementation? I also recently implemented NP (in PyTorch), but I can't get it to work for that task. After seeing your code here, I made a version of my own code that should be ~identical to yours (except in PyTorch). I only did your first experiment, but it worked fine. However, the image completion works no better than my previous implementation. I'd be interested to hear about any results that you had on that task / thoughts on what could be failing there (feel free to close this issue if you aren't interested).

Sorry this question might be obvious but I don't understand why n_draws and N_star are taken from [1] from shape. Shouldn't their dimension be taken from [0].

decoder g -- map (z, x*) -> hidden -> y*
-inputs dimensions
-z_sample has dim [n_draws, dim_z]

• x_star has dim [N_star, dim_x]

n_draws <- z_sample$get_shape()$as_list()[1]
N_star <- tf$shape(x_star)[1]