Inpainting using Denoising Diffusion Probabilistic Models [Paper]

Star this project to get notified.

What are the blue parts?
Those parts are missing and therefore have to be filled by RePaint.
RePaint generates the missing parts inspired by the known parts.

How does it work?
RePaint starts from pure noise. Then the image is denoised step-by-step.
It uses the known part to fill the unknown part in each step.

Why does the noise level fluctuate during generation?
Our noise schedule improves the harmony between the generated and
the known part [4.2 Resampling].

The blue region is unknown and filled by RePaint:

Note: RePaint creates many meaningful fillings.

1. Face: Expressions and features like an earring or a mole.
   
2. Computer: The computer screen shows different images, text, and even a logo.
   
3. Greens: RePaint makes sense of the tiny known part and incorporates it in a beetle, spaghetti, and plants.
   
4. Garden: From simple filling like a curtain to complex filling like a human.
   

• Every Second line of the input image is unknown.
• Most inpainting methods fail on such masks.

• The inpainting only knows pixels with a strided access of 2.
• A ratio of 3/4 of the image has to be filled.
• This is equivalent to Super-Resolution with Nearest Neighbor kernel.

• RePaint uses Denoising Diffusion Probabilistic Models.
• We condition this process on the given image content.

Intuition of one denoising step:

1. Sample the known part: Add gaussian noise to the known regions of the image.
   We obtain a noisy image that follows the denoising process exactly.
2. Denoise one step: Denoise the previous image for one step. This generates
   content for the unknown region conditioned on the known region.
3. Join: Merge the images from both steps.

Details are in Algorithm 1 on Page 4. [Paper]

• Fail: When using only the algorithm above, the filling is not well harmonized with the known part (n=1).
• Fix: When applying the [4.2 Resampling] technique, the images are better harmonized (n>1).

• The ImageNet model is biased towards inpainting dogs.
• This is due to the high ratio of dog images in ImageNet.

• Outperforms autoregressive-based and GAN-based SOTA methods for
  all masks with significance 95% except for two inconclusive cases.
• The user study was done for six different masks on three datasets.
• RePaint outperformed SOTA methods in 42 of 44 cases. [Paper]

• Datasets: CelebA-HQ, ImageNet, Places2
• Masks: Random strokes, half image, huge, sparse
• Explore more examples like this in the [Appendix].

Star this project to get notified.