Light

ai-forever / movqgan Goto Github PK

View Code? Open in Web Editor NEW

100.0 4.0 7.0 28.11 MB

MoVQGAN - model for the image encoding and reconstruction

Python 5.51% Jupyter Notebook 94.49%

gan generative-adversarial-network image-compression image-encoding image-reconstruction

movqgan's Introduction

SBER-MoVQGAN

SBER-MoVQGAN (Modulated Vector Quantized GAN) is a new SOTA model in the image reconstruction problem. This model is based on code from the VQGAN repository and modifications from the original MoVQGAN paper. The architecture of SBER-MoVQGAN is shown below in the figure.

SBER-MoVQGAN was successfully implemented in Kandinsky 2.1, and became one of the architecture blocks that allowed to significantly improve the quality of image generation from text.

Models

The following table shows a comparison of the models on the Imagenet dataset in terms of FID, SSIM, and PSNR metrics. A more detailed description of the experiments and a comparison with other models can be found in the Habr post.

Model	Latent size	Num Z	Train steps	FID	SSIM	PSNR	L1
ViT-VQGAN*	32x32	8192	500000	1,28	-	-	-
RQ-VAE*	8x8x16	16384	10 epochs	1,83	-	-	-
Mo-VQGAN*	16x16x4	1024	40 epochs	1,12	0,673	22,42	-
VQ CompVis	32x32	16384	971043	1,34	0,65	23,847	0,053
KL CompVis	32x32	-	246803	0,968	0,692	25,112	0,047
SBER-VQGAN (from pretrain)	32x32	8192	1 epoch	1,439	0,682	24,314	0,05
SBER-MoVQGAN 67M	32x32	16384	2M	0,965	0,725	26,449	0,042
SBER-MoVQGAN 102M	32x32	16384	2360k	0,776	0,737	26,889	0,04
SBER-MoVQGAN 270M	32x32	16384	1330k	0,686💥	0,741💥	27,037💥	0,039💥

How to use

Install

pip install "git+https://github.com/ai-forever/MoVQGAN.git"

Train

python main.py --config configs/movqgan_270M.yaml

Inference

Check jupyter notebook with example in ./notebooks folder or

Examples

This section provides examples of image reconstruction for all versions of SBER-MoVQGAN on hard-to-recover domains such as faces, text, and other complex scenes.

Authors

Anastasia Maltseva: Github
Arseniy Shakhmatov: Github, Blog
Andrey Kuznetsov: Github, Blog
Denis Dimitrov: Github, Blog

movqgan's People

Contributors

Stargazers

Watchers

Forkers

echo-cwb quickgrid iejmac jpfeil chengzhoulee bruinxiong

movqgan's Issues

About more detailed configuration for model reproduction

Awesome work!
However, I somehow encountered the difficulty reproducing the performance of the autoencoder by training-from-scratch. Especially with the addition of GAN loss, getting collapsed reconstruction results.
I used a subset of laion with aesthetic score >=5, training with batch size=256 and default loss weighting configurations. May I ask the appropriate configuration for reproducing the performance of MoVQGAN? Thank you very much.

about fp16 version

Admiring work! Do you have a fp16 version which would be better for other works like text-to-image?

cuda out of memory

I modified the 67M version configs to train the network with 384*384 imgs, the size of z changed to 48*48*4. However it shown "cuda out of memory", even on an A100*80G GPU

Ablation about n_embed and embed_dim?

Hello, may I ask you how does the hyperparameters n_embed and embed_dim influence the performance, could you give me some insight

Is perceptual loss using a random initialized VGG?

Hi, thanks for sharing the code. Here seems that you are using a pretrained=False VGG network. I wonder if this is the common practice? Thanks

Question about the learning environment for another dataset

I want to training from a small size image like cifar-10, wondering how to do it.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.