sahajgarg / image_transformer Goto Github PK

Hi, Sahaj. Maybe a dumb question. Can I know how the masked attention weigtht is implemented in your script?

Hi there, I essentially have the same problem as this issue. Would you mind clarifying how masking is implemented?

I want to modify masking so that I may get inference to work in a different order.

Very nice job, the code is very nice!

Context. Autoregressive models (including Image Transformer) can be viewed as a one layer normalizing flow as described in this article Section 4. The article contains code which trains PixelCNN as a flow. I am trying to do the same for Image Transformer by using your implementation.

Question 1. PixelCNN generates in raster scan order. Does your Pixel Transformer implementation also use raster scan order?

Hi,

Thanks for sharing your codes. But it seems there are two problems with your code. Please ignore it if I am wrong.

It seems you forget to add position embedding to the input representations. Maybe your add_timing_signal finish this job, but I am not sure. By the way, what is add_timing_signal function for?

In your training code, you forget to generate a mask to mask off the pixels that have not been generated. It means the previous pixel can get the information of the next pixel.