dave-epstein / blobgan Goto Github PK

View Code? Open in Web Editor NEW

170.0 170.0 11.0 136 KB

Official PyTorch implementation of BlobGAN: Spatially Disentangled Scene Representations

License: BSD 2-Clause "Simplified" License

Python 95.82% C++ 0.46% Cuda 3.72%

blobgan's People

Contributors

Stargazers

Watchers

Forkers

osifa78 doantientai tanglespace nijsf lime-cakes ai-hub-deep-learning-fundamental ystyldod phillipinseoul alexander-novo brandonlucasbytes masixian

blobgan's Issues

Pretrained encoder

Thank you for providing the code. Your work is very interesting.
Do you intend to provide the weights of the encoder for image inversion? on bedroom for example?
Thank you

why pass a dict to layoutstyledGan as styles parmater?

blobgan/src/models/blobgan.py

Line 185 in 886a44b

'styles': {k: layout[k] for k in SPLAT_KEYS} if self.spatial_style else z,

Getting Error of "No Image" When Trying to Train on Custom Dataset

Hi. I was trying to replicate the experiment with my custom dataset of bedrooms. I've modified my local.yaml and set the basepath to the image folder, where it contains train, test, and validate. I'm sure that they all contain images. But when I execute this:

python src/run.py +experiment=[blobgan,local,jitter] wandb.name='10-blob BlobGAN on bedrooms'

It gives this error:

AssertionError: Cannot compute FID without name of statistics file to use.

After scrolling up, I found these errors:

Warning: no images found in /home/PythonProjects/blobgan/resizedout/train. Using empty dataset for split train. Perhaps you set `dataset.path` incorrectly?
Warning: no images found in /home/PythonProjects/blobgan/resizedout/validate. Using empty dataset for split validate. Perhaps you set `dataset.path` incorrectly?
Warning: no images found in /home/PythonProjects/blobgan/resizedout/test. Using empty dataset for split test. Perhaps you set `dataset.path` incorrectly?

This is the entire error log:

resume:
  id: null
  step: null
  epoch: null
  last: true
  best: false
  clobber_hparams: false
  project: Blobgan investigation
  log_dir: ./logs
  model_only: false
logger: wandb
wandb:
  save_code: true
  offline: false
  log_dir: ./logs
  id: null
  name: 10-blob BlobGAN on bedrooms
  group: XXX Group
  project: Blobgan investigation
  entity: yimingsu
trainer:
  accelerator: ddp
  benchmark: false
  deterministic: false
  gpus: 1
  precision: 32
  plugins: null
  max_steps: 10000000
  profiler: simple
  num_sanity_val_steps: 0
  log_every_n_steps: 200
  limit_val_batches: 0
dataset:
  dataloader:
    prefetch_factor: 2
    pin_memory: true
    drop_last: true
    persistent_workers: true
    num_workers: 12
    batch_size: 24
  name: ImageFolderDataModule
  resolution: 256
  category: bedroom
  path: /home/yimingsu/PythonProjects/blobgan/resizedout/
mode: fit
seed: 0
checkpoint:
  every_n_train_steps: 1500
  save_top_k: -1
  mode: max
  monitor: step
model:
  name: BlobGAN
  lr: 0.002
  dim: 512
  noise_dim: 512
  resolution: 256
  lambda:
    D_real: 1
    D_fake: 1
    D_R1: 50
    G: 1
    G_path: 2
    G_feature_mean: 10
    G_feature_variance: 10
  discriminator:
    name: StyleGANDiscriminator
    size: 256
  generator:
    name: models.networks.layoutstylegan.LayoutStyleGANGenerator
    style_dim: 512
    n_mlp: 8
    size_in: 16
    c_model: 96
    spatial_style: true
    size: 256
  layout_net:
    name: models.networks.layoutnet.LayoutGenerator
    n_features_max: 10
    feature_dim: 768
    style_dim: 512
    noise_dim: 512
    norm_features: true
    mlp_lr_mul: 0.01
    mlp_hidden_dim: 1024
    spatial_style: true
  D_reg_every: 16
  G_reg_every: -1
  λ:
    D_real: 1
    D_fake: 1
    D_R1: 50
    G: 1
    G_path: 2
    G_feature_mean: 10
    G_feature_variance: 10
  log_images_every_n_steps: 1000
  n_features_min: 10
  n_features_max: 10
  n_features: 10
  spatial_style: true
  feature_jitter_xy: 0.04
  feature_jitter_shift: 0.5
  feature_jitter_angle: 0.1

Global seed set to 0
wandb: Currently logged in as: yimingsu (use `wandb login --relogin` to force relogin)
wandb: wandb version 0.13.2 is available!  To upgrade, please run:
wandb:  $ pip install wandb --upgrade
wandb: Tracking run with wandb version 0.12.11
wandb: Run data is saved locally in /home/yimingsu/PythonProjects/blobgan/logs/wandb/run-20220902_114754-xjimkxit
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run 10-blob BlobGAN on bedrooms
wandb: ⭐️ View project at https://wandb.ai/yimingsu/Blobgan%20investigation
wandb: 🚀 View run at https://wandb.ai/yimingsu/Blobgan%20investigation/runs/xjimkxit
[2022-09-02 11:47:58,898][torch.distributed.nn.jit.instantiator][INFO] - Created a temporary directory at /tmp/tmpr5cdgp32
[2022-09-02 11:47:58,898][torch.distributed.nn.jit.instantiator][INFO] - Writing /tmp/tmpr5cdgp32/_remote_module_non_sriptable.py
Froze 65 parameters - ['conv1.conv', 'conv1.noise', 'conv1.activate', 'to_rgb1', 'to_rgb1.conv', 'convs.0', 'convs.1', 'convs.2', 'convs.3', 'convs.4', 'convs.5', 'convs.6', 'convs.7', 'to_rgbs.0', 'to_rgbs.1', 'to_rgbs.2', 'to_rgbs.3'] - for model of type LayoutStyleGANGenerator
Froze 16 parameters - ['mlp.1', 'mlp.2', 'mlp.3', 'mlp.4', 'mlp.5', 'mlp.6', 'mlp.7', 'mlp.8'] - for model of type LayoutGenerator
[2022-09-02 11:47:59,637][py.warnings][WARNING] - /home/yimingsu/anaconda3/envs/blobgan/lib/python3.9/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:286: LightningDeprecationWarning: Passing `Trainer(accelerator='ddp')` has been deprecated in v1.5 and will be removed in v1.7. Use `Trainer(strategy='ddp')` instead.
  rank_zero_deprecation(

[2022-09-02 11:47:59,637][py.warnings][WARNING] - /home/yimingsu/anaconda3/envs/blobgan/lib/python3.9/site-packages/pytorch_lightning/trainer/connectors/callback_connector.py:147: LightningDeprecationWarning: Setting `Trainer(checkpoint_callback=True)` is deprecated in v1.5 and will be removed in v1.7. Please consider using `Trainer(enable_checkpointing=True)`.
  rank_zero_deprecation(

GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
Global seed set to 0
initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/1
[2022-09-02 11:47:59,641][torch.distributed.distributed_c10d][INFO] - Added key: store_based_barrier_key:1 to store for rank: 0
[2022-09-02 11:47:59,641][torch.distributed.distributed_c10d][INFO] - Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
----------------------------------------------------------------------------------------------------
distributed_backend=nccl
All distributed processes registered. Starting with 1 processes
----------------------------------------------------------------------------------------------------

Warning: no images found in /home/yimingsu/PythonProjects/blobgan/resizedout/train. Using empty dataset for split train. Perhaps you set `dataset.path` incorrectly?
Warning: no images found in /home/yimingsu/PythonProjects/blobgan/resizedout/validate. Using empty dataset for split validate. Perhaps you set `dataset.path` incorrectly?
Warning: no images found in /home/yimingsu/PythonProjects/blobgan/resizedout/test. Using empty dataset for split test. Perhaps you set `dataset.path` incorrectly?
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
Optimizing 57.19M params for G and 28.86M params for D

  | Name           | Type                    | Params
-----------------------------------------------------------
0 | discriminator  | StyleGANDiscriminator   | 28.9 M
1 | generator_ema  | LayoutStyleGANGenerator | 35.9 M
2 | generator      | LayoutStyleGANGenerator | 35.9 M
3 | layout_net_ema | LayoutGenerator         | 21.3 M
4 | layout_net     | LayoutGenerator         | 21.3 M
-----------------------------------------------------------
86.1 M    Trainable params
57.2 M    Non-trainable params
143 M     Total params
573.008   Total estimated model params size (MB)
[2022-09-02 11:48:01,076][py.warnings][WARNING] - /home/yimingsu/anaconda3/envs/blobgan/lib/python3.9/site-packages/torch/utils/data/dataloader.py:487: UserWarning: This DataLoader will create 12 worker processes in total. Our suggested max number of worker in current system is 8, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  warnings.warn(_create_warning_msg(

Training: 0it [00:00, ?it/s]Error executing job with overrides: ['+experiment=[blobgan,local,jitter]', 'wandb.name=10-blob BlobGAN on bedrooms', 'dataset=imagefolder', '++dataset.path=/home/yimingsu/PythonProjects/blobgan/resizedout/']
Traceback (most recent call last):
  File "/home/yimingsu/PythonProjects/blobgan/src/run.py", line 81, in run
    trainer.fit(model, datamodule=datamodule)
  File "/home/yimingsu/anaconda3/envs/blobgan/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 740, in fit
    self._call_and_handle_interrupt(
  File "/home/yimingsu/anaconda3/envs/blobgan/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 685, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/yimingsu/anaconda3/envs/blobgan/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 777, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/home/yimingsu/anaconda3/envs/blobgan/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1199, in _run
    self._dispatch()
  File "/home/yimingsu/anaconda3/envs/blobgan/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1279, in _dispatch
    self.training_type_plugin.start_training(self)
  File "/home/yimingsu/anaconda3/envs/blobgan/lib/python3.9/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 202, in start_training
    self._results = trainer.run_stage()
  File "/home/yimingsu/anaconda3/envs/blobgan/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1289, in run_stage
    return self._run_train()
  File "/home/yimingsu/anaconda3/envs/blobgan/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1319, in _run_train
    self.fit_loop.run()
  File "/home/yimingsu/anaconda3/envs/blobgan/lib/python3.9/site-packages/pytorch_lightning/loops/base.py", line 140, in run
    self.on_run_start(*args, **kwargs)
  File "/home/yimingsu/anaconda3/envs/blobgan/lib/python3.9/site-packages/pytorch_lightning/loops/fit_loop.py", line 200, in on_run_start
    self.trainer.call_hook("on_train_start")
  File "/home/yimingsu/anaconda3/envs/blobgan/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1501, in call_hook
    output = model_fx(*args, **kwargs)
  File "/home/yimingsu/PythonProjects/blobgan/src/models/blobgan.py", line 131, in on_train_start
    assert not ((self.log_fid_every_n_steps > -1 or self.log_fid_every_epoch) and (not self.fid_stats_name)), \
AssertionError: Cannot compute FID without name of statistics file to use.

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

Any help would be greatly appreciated!

Best,
Yiming

Restyling and cloning code

Thanks for your great work and well-organized code! Could you please also share the restyling and cloning code? I tried restyling, but I'm not sure what's the thin border of the blob; I also tried cloning by simply copying all the parameters of one blob, but I can't get the same object on the image.

Getting "RuntimeError: CUDA error: out of memory" when trying to train on multiple GPUs

Hi,

I'm getting this error when trying to train my own model on multiple GPUs.

This is my command:
python src/run.py +experiment=[blobgan,local,jitter] wandb.name='10-blob BlobGAN on bedrooms

This is my local.yaml:

dataset:
  path: /home/yimingsu/blobgan/resizedout # Change to your path
  resolution: 128
  #dataloader:
    #batch_size: 24
trainer:
  gpus: 4  # Change to your number of GPUs
wandb:  # Fill in your settings
  group:  Group
  project: Blobgan investigation
  entity: yimingsu
model:
  #layout_net:
    #feature_dim: 256
  #generator:
    #override_c_in: 256
  #dim: 256
  fid_stats_name: mitbedroom # this is my custom dataset.

This is the error log:

Error executing job with overrides: ['+experiment=[blobgan,local,jitter]', 'wandb.name=10-blob BlobGAN on bedrooms']
Traceback (most recent call last):
  File "/home/yimingsu/blobgan/src/run.py", line 81, in run
    trainer.fit(model, datamodule=datamodule)
  File "/home/yimingsu/blobganenv/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 740, in fit
    self._call_and_handle_interrupt(
  File "/home/yimingsu/blobganenv/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 685, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/yimingsu/blobganenv/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 777, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/home/yimingsu/blobganenv/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1138, in _run
    self._call_setup_hook()  # allow user to setup lightning_module in accelerator environment
  File "/home/yimingsu/blobganenv/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1435, in _call_setup_hook
    self.training_type_plugin.barrier("pre_setup")
  File "/home/yimingsu/blobganenv/lib/python3.9/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 403, in barrier
    torch.distributed.barrier(device_ids=self.determine_ddp_device_ids())
  File "/home/yimingsu/blobganenv/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 2784, in barrier
    work = default_pg.barrier(opts=opts)
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

I'm training on 4 RTX 2080 Ti so I don't think the memory is the actual issue.

Any help would be appreciated!

How to edit blobs?

Hi,

Thanks for the repo. It is pretty interesting to play around.

I am wondering how do I edit the blobs to make things vanish? Presumably, I will be editing the outputs from the inversion models, right? So, how should I do that?

Best,
Jian

Changing number of blobs

In appendix E.1 Hyperparameters of your paper you mentioned that you used different model configurations depending on the dataset (number of objects per image).
I want to train a model with k=40, d_in= 256 and d_style=256. I changed output shape of the layout network according to these values and 'c_model' of the generator to 32 to match to spatial_style number of channels.

This is my config file:

model:
name: BlobGAN
lr: 0.002
dim: 256
noise_dim: 512
resolution: 256
lambda:
D_real: 1
D_fake: 1
D_R1: 50
G: 1
G_path: 2
G_feature_mean: 10
G_feature_variance: 10
discriminator:
name: StyleGANDiscriminator
size: ${model.resolution}
generator:
name: models.networks.layoutstylegan.LayoutStyleGANGenerator
style_dim: ${model.dim}
n_mlp: 8
size_in: 16
c_model: 32
spatial_style: ${model.spatial_style}
size: ${model.resolution}
layout_net:
name: models.networks.layoutnet.LayoutGenerator
n_features_max: ${model.n_features_max}
feature_dim: 256
style_dim: ${model.dim}
noise_dim: ${model.noise_dim}
norm_features: true
mlp_lr_mul: 0.01
mlp_hidden_dim: 1024
spatial_style: ${model.spatial_style}
n_features_min: ${model.n_features}
n_features_max: ${model.n_features}
n_features: 40
spatial_style: true

Hower with this configuration I get very few parameters in the generator:

61.7 M Trainable params
32.9 M Non-trainable params
94.6 M Total params
378.321 Total estimated model params size (MB)

Cannot reproduce a layout

Hi,

I cannot reproduce with the same layout

layout, img = model.gen(z, ret_layout=True)
layout, img = model.gen(layout=layout, ret_layout=True)

blobgan/src/models/blobgan.py

Line 303 in c527f1c

covs: [N, M, 2, 2] xy covariance matrices for each feature

blobgan/src/models/blobgan.py

Line 323 in c527f1c

a, b = covs[..., :2].sigmoid().unbind(-1)

I am wondering the basis computation is wrong here, which you will get a tensor with shape BxCx2x0:

blobgan/src/models/blobgan.py

Line 327 in c527f1c

basis_i = covs[..., 2:]

Regarding the running time

Hi.

While reading the paper, I'm a little bit confused of the training time for the BlobGAN.

Specifically, in the Appendix E.1 in the paper, it is written as "We train all models for 1.5 million gradient steps with batch size 24 per-GPU across 8 NVIDIA A100 GPUs. This takes around 4 weeks."

Now I'm wondering if it takes 4 weeks with 8 NVIDIA A100 GPUs for training a single model or for all models.

Could you please clarify this more?

Thanks!

Inversion

Hi,

I have a question regarding image inversion. To train the encoder, did you use the jittering of the spatial parameters? In the 'invertblobgan.py' code no_jitter is not used and the invertblobgan.yaml doesn't set the jittering parameters to 0. Is this intended?

Colab Interactively edit scenes error (indices should be either on cpu or on the same device as the indexed tensor (cpu))

Hi Dr. Epstein,

Thank you for this great work and published codes.

I tried to manipulate blobs using your BlobGAN Playground Colab (Last cell: Interactively edit scenes). I also ran the code on the server.

They both gave the same error message

RuntimeError Traceback (most recent call last)
in <cell line: 37>()
35 layout, orig_img = model.gen(z=z, truncate=truncate, layout=layout, **render_kwargs)
36 orig_blobs = for_canvas(layout['feature_img'].mul(255))
---> 37 labeled_blobs, labeled_blobs_img = draw_labels(orig_blobs, layout, size_threshold, COLORS)
38
39 blobs = DraggableBlobMap(locals())

/content/blobgan/./src/utils/colab.py in draw_labels(img, layout, T, colors, layout_i)
88 # draw = ImageDraw.Draw(img)
89 # mask = layout['sizes'][layout_i, 1:] > T
---> 90 # idmap = torch.arange(len(mask))[mask]
91 # blob = {k: layout[k][layout_i][mask].mul(255).tolist() for k in ('xs', 'ys')}
92 # for i, (x, y) in enumerate(zip(blob['xs'], blob['ys'])):

RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)

I searched online and no solutions work for me. Could you please tell me how to solve this for manipulating blobs and generating the corresponding images?

I appreciate any help from you.

Blob size and opacity

Hello,
I have some questions regarding paragraph 3.1 of the paper:

have you tried to make the constant 'c' controlling blob edge sharpness a trainable parameter?
correct me if I am wrong, but when 's' is positive, the maximum opacity of the blob is not located at its centroid but at a distance s from it. Is this coorect? Is it the expected behaviour? Does the model learn only negative sizes to avoid this?
Blobs size and opacity are controlled with the same parameter 's', so when we increase 's_k' of a blob k when editing an image we are increasing its size as well as its maximum opacity ==> this will change score values even at the center of the blob ==> more weights to the style vector of blob k ==> style in that region will change too. Haven't you tried to model size and maximum opacity using two different parameters? eg. modeling the opacity as a gaussian function centred at the centroid (x,y) with a standard deviation 's' cotrolling its size and multiplied by a scale factor 'm' controlling the maximum opacity of the blob
$$
o(x_{grid}) = m \sigma_{s}(d(x_{grid},x)
$$

Best,
Mehdi

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.