advimman / cips Goto Github PK

Thanks for sharing nice implementation!
Could you explain how is the initialization constant being chosen in LFF block?

https://github.com/saic-mdal/CIPS/blob/eadae6e45d8c1f3657faa88a065b59990747cd16/model/blocks.py#L517

Can you please explain what this does in the notebook? Should this truncation be recomputed if one is to create more diversity of generation?

/home/hello/miniconda3/envs/deepsort/bin/python3 /home/hello/桌面/CIPS-main/train.py LMDB_data1024*1024_anime --nproc_per_node=8 --master_port=1234 --n_sample=8 --batch=4 --fid_batch=8 --Generator=CIPSskip --output_dir=skip-[ffhq/churches] --img2dis --num_workers=16
Traceback (most recent call last):
File "/home/hello/桌面/CIPS-main/train.py", line 16, in
import model
File "/home/hello/桌面/CIPS-main/model/init.py", line 1, in
from .Discriminators import *
File "/home/hello/桌面/CIPS-main/model/Discriminators.py", line 8, in
from .blocks import ConvLayer, ResBlock, EqualLinear
File "/home/hello/桌面/CIPS-main/model/blocks.py", line 8, in
from op import FusedLeakyReLU, fused_leaky_relu, upfirdn2d
File "/home/hello/桌面/CIPS-main/op/init.py", line 2, in
from .upfirdn2d import upfirdn2d
File "/home/hello/桌面/CIPS-main/op/upfirdn2d.py", line 13, in
os.path.join(module_path, 'upfirdn2d_kernel.cu'),
File "/home/hello/miniconda3/envs/deepsort/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1092, in load
keep_intermediates=keep_intermediates)
File "/home/hello/miniconda3/envs/deepsort/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1307, in _jit_compile
baton.wait()
File "/home/hello/miniconda3/envs/deepsort/lib/python3.7/site-packages/torch/utils/file_baton.py", line 42, in wait
time.sleep(self.wait_seconds)
KeyboardInterrupt

Process finished with exit code 1

Hello authors,

Thank you for your paper -- it looks very interesting! I had a quick question about training time: how long (and with how many GPUs) does it take you to train the pretrained models provided in the repo? Do you find that the method is slow or fast relative to StyleGAN2?

I look forward to playing around with the repo over the next week!

Best,
greeneggsandyaml

Dear authors:

Hello, thank you for sharing the great project.

I would like to use your project for the baseline of my future research.

Would you please share your discriminator's weight also?

Thank you.

Best,
chkimmmmm

Hi. First of all, thanks for your great work.

I have a problem when calculating FID during training. Every args.save_checkpoint_frequency iterations, the model is evaluated by calculating FID score. However, at this phase, I have a timed out problem. Here is the error log.

[E ProcessGroupNCCL.cpp:566] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1802801 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:325] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.
terminate called after throwing an instance of 'std::runtime_error'
  what():  [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1802801 milliseconds before timing out.
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 1 (pid: 4414) of binary: /opt/conda/bin/python
ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed

Before this log came out, there was no change in the prompt but GPUs were still working. As I'm not that familiar with distributed training, I want to ask how to fix this problem. I've tried to lower the number of args.fid_samples, but this doesn't help. Thank you.

Best Wishes,
Lee

Hi!
Could you please specify, how you mine the Satellite-Landscapes dataset?
Best regards, Andrey.

It seems you are passing the path argument to args.out_path (line 332 in train.py). Is that the normal procedure?

Should that be commented out, in order to use the path argument in Multiscale dataset? Or is it expected that the dataset folders are inside the output_path?

Thanks!

Dear authors:

Hello.

First of all, thank you for the sharing this great work.

In the Read.me, you mentioned that you are going to share the progressive training details.

When will you share it?

Thank you.

Best.
chkimmmmm

Hi,

Really nice work! I wonder if you have checkpoints for models in the ablation study? For example, models trained without the coordinate embedding? THanks!

Why do you have an generator and g_ema in your training code？And Your optimizer optimizes generator directly，why not optimizes g_ema? And why accumulate(g_ema, g_module, accum) ，not And why accumulate(g_ema, generator, accum) ？

Hi, excellent work!

In your paper, you discuss a Landscapes dataset of FLICKR images that are randomly cropped to 256x256 for training. Figure 15 also shows results for patch-based training on Churches and FFHQ. How many coordinate embeddings are learned in the patch-based training settings? Are there patch_height x patch_width learned embeddings, or full_res_image_height x full_res_image_width embeddings? If the former, do you tile the patch embedding grids when synthesizing panoramas?

Hello,

Thanks for the code! Trying to run my first experiment but it seems that the default value for the generator is not available. I believe right now the choice is between CIPSskip and CIPSres.

The train.py command works with the above options.

I see that in the code for CIPS block, you use a Blur layer, which use 3x3 blur kernel so I think one pixel is affect by 8 surrouding pixel, is it correct ?