wanggcong / stylelight Goto Github PK

Hi!

I notice that you may use Virtual Object Relighting (VOR) dataset to evaluation qualitatively because i found you ask some question about this in the issue of EMLight, and there are some crops in the './StyleLight/assets/wild2/' folders.

I am trying to make Qualitative Evaluation, too. But i could not found the ground truth panaroma picture in their Virtual Object Relighting (VOR) dataset. So i want to ask you that have you found GT panaroma in VOR dataset? If you GT panaroma were not included in VOR, can you share this GT panaromas with me?

Thanks for sharing you excelent work！

It is a great honor for me to study your results.After I estimate the HDR panorama with the model, I use it to render virtual objects in blender.
I've noticed that rendering directly from the output results in very dark renders. I had to multiply the light intensity by a factor of 100 to get a visually ideal rendering.

I would like to know what are the reasons for this? Looking forward to your reply!

I would like to thank you for creating such a great work.

I'm currently working on light estimation and want to compare it with the Stylelight.

However, when I ran your code on the Laval indoor dataset, the score is quite different. So, I would like to ask if this is still acceptable. If not, I am looking for the correct way to run an evaluation.

Let me specify how I got this score so you can point out which step I did wrong.

1. Prepare the cropped Laval indoor dataset by running data_prepare_laval.py
2. changing input path by pointing root_path in test_lighting.py to the corrected path
3. Tonemap the ground truth in the directory name test using evaluation/tonemap.py
4. Tonemap the output of Stylelight using evaluation/tonemap.py
5. Render ground truth into 3 balls using evaluation/test_render.sh
6. Render the output of Stylelight into 3 balls using evaluation/test_render.sh
7. Compute the score by run evaluation/test_rmse.sh --fake <stylelight's tone-mapped dir> --real <ground truth's tone-mapped dir>

Best regard.

hi,I want to know where is the code for calculating MSE and RMSE？

Hi!
Thanks for your amazing work.
When I try to run python test_lighting.py, I got error above from demo_crop.py. It seems you have modified the origin skylibs. Could you please provide the implementation of this function? Or where can I get the function 'Fov2MaskedPano'? Thanks!

hi,When I was training with a dataset, I encountered the following question. If I could get your answer, thank you very much！

:~/StyleLight-main$ CUDA_VISIBLE_DEVICES=3,4 python train.py --outdir=./training-runs-256x512 --data=/home/kf/LavalIndoorDataset/IndoorHDRDataset2018-debug2-256x512-data-splits2/train/IndoorHDRDataset2018 --gpus=2
batch size: 32
batch gpu: 16

Training options:
{
"num_gpus": 2,
"image_snapshot_ticks": 100,
"network_snapshot_ticks": 100,
"metrics": [
"fid50k_full"
],
"random_seed": 0,
"training_set_kwargs": {
"class_name": "training.dataset.ImageFolderDataset",
"path": "/home/kf/LavalIndoorDataset/IndoorHDRDataset2018-debug2-256x512-data-splits2/train/IndoorHDRDataset2018",
"use_labels": false,
"max_size": 255,
"xflip": false,
"resolution": 256
},
"data_loader_kwargs": {
"pin_memory": true,
"num_workers": 3,
"prefetch_factor": 2
},
"G_kwargs": {
"class_name": "training.networks.Generator",
"z_dim": 512,
"w_dim": 512,
"mapping_kwargs": {
"num_layers": 2
},
"synthesis_kwargs": {
"channel_base": 16384,
"channel_max": 512,
"channels_dict": {
"4": 512,
"8": 512,
"16": 512,
"32": 512,
"64": 512,
"128": 256,
"256": 128,
"512": 64
},
"num_fp16_res": 4,
"conv_clamp": 256
}
},
"D_kwargs": {
"class_name": "training.networks.Discriminator",
"block_kwargs": {},
"mapping_kwargs": {},
"epilogue_kwargs": {
"mbstd_group_size": 4
},
"channel_base": 16384,
"channel_max": 512,
"channels_dict": {
"4": 512,
"8": 512,
"16": 512,
"32": 512,
"64": 512,
"128": 256,
"256": 128,
"512": 64
},
"num_fp16_res": 4,
"conv_clamp": 256
},
"task_name": "StyleLight-training",
"G_opt_kwargs": {
"class_name": "torch.optim.Adam",
"lr": 0.0025,
"betas": [
0,
0.99
],
"eps": 1e-08
},
"D_opt_kwargs": {
"class_name": "torch.optim.Adam",
"lr": 0.0025,
"betas": [
0,
0.99
],
"eps": 1e-08
},
"loss_kwargs": {
"class_name": "training.loss.StyleGAN2Loss",
"r1_gamma": 0.4096
},
"total_kimg": 25000,
"batch_size": 32,
"batch_gpu": 16,
"ema_kimg": 10.0,
"ema_rampup": 0.05,
"ada_target": 0.6,
"augment_kwargs": {
"class_name": "training.augment.AugmentPipe",
"xflip": 1,
"rotate90": 1,
"xint": 1,
"scale": 1,
"rotate": 1,
"aniso": 1,
"xfrac": 1,
"brightness": 1,
"contrast": 1,
"lumaflip": 1,
"hue": 1,
"saturation": 1
},
"run_dir": "./training-runs-256x512/00002-IndoorHDRDataset2018-auto2"
}

Output directory: ./training-runs-256x512/00002-IndoorHDRDataset2018-auto2
Training data: /home/kf/LavalIndoorDataset/IndoorHDRDataset2018-debug2-256x512-data-splits2/train/IndoorHDRDataset2018
Training duration: 25000 kimg
Number of GPUs: 2
Number of images: 255
Image resolution: 256
Conditional model: False
Dataset x-flips: False

Creating output directory...
Launching processes...
Loading training set...
training_set_kwargs: {'class_name': 'training.dataset.ImageFolderDataset', 'path': '/home/kf/LavalIndoorDataset/IndoorHDRDataset2018-debug2-256x512-data-splits2/train/IndoorHDRDataset2018', 'use_labels': False, 'max_size': 255, 'xflip': False, 'resolution': 256}

Num images: 255
Image shape: [6, 256, 512]
Label shape: [0]

Constructing networks...
Setting up augmentation...
Distributing across 2 GPUs...
Setting up training phases...
Exporting sample images...

...........................

Traceback (most recent call last):
File "train.py", line 581, in
main() # pylint: disable=no-value-for-parameter
File "/home/kf/miniconda3/envs/stylegan2/lib/python3.7/site-packages/click/core.py", line 1130, in call
return self.main(*args, **kwargs)
File "/home/kf/miniconda3/envs/stylegan2/lib/python3.7/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/home/kf/miniconda3/envs/stylegan2/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/kf/miniconda3/envs/stylegan2/lib/python3.7/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/home/kf/miniconda3/envs/stylegan2/lib/python3.7/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "train.py", line 576, in main
torch.multiprocessing.spawn(fn=subprocess_fn, args=(args, temp_dir), nprocs=args.num_gpus)
File "/home/kf/miniconda3/envs/stylegan2/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 199, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/home/kf/miniconda3/envs/stylegan2/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 157, in start_processes
while not context.join():
File "/home/kf/miniconda3/envs/stylegan2/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 118, in join
raise Exception(msg)
Exception:

-- Process 1 terminated with the following error:
Traceback (most recent call last):
File "/home/kf/miniconda3/envs/stylegan2/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
fn(i, *args)
File "/home/kf/StyleLight-main/train.py", line 427, in subprocess_fn
training_loop.training_loop(rank=rank, **args)
File "/home/kf/StyleLight-main/training/training_loop.py", line 352, in training_loop
loss.accumulate_gradients(phase=phase.name, real_img=real_img, real_c=real_c, gen_z=gen_z, gen_c=gen_c, sync=sync, gain=gain)
File "/home/kf/StyleLight-main/training/loss.py", line 99, in accumulate_gradients
gen_logits = self.run_D(gen_img_ldr, gen_c, sync=False,isRealImage=False) ######## add isRealImage=False
File "/home/kf/StyleLight-main/training/loss.py", line 69, in run_D
logits = self.D(img, c)
File "/home/kf/miniconda3/envs/stylegan2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/kf/miniconda3/envs/stylegan2/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 619, in forward
output = self.module(*inputs[0], **kwargs[0])
File "/home/kf/miniconda3/envs/stylegan2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/kf/StyleLight-main/training/networks.py", line 755, in forward
x, img = block(x, img, **block_kwargs)
File "/home/kf/miniconda3/envs/stylegan2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(input, **kwargs)
File "/home/kf/StyleLight-main/training/networks.py", line 580, in forward
misc.assert_shape(x, [None, self.in_channels, self.resolution, 2self.resolution])
File "/home/kf/StyleLight-main/torch_utils/misc.py", line 97, in assert_shape
raise AssertionError(f'Wrong size for dimension {idx}: got {size}, expected {ref_size}')
AssertionError: Wrong size for dimension 2: got 64, expected 128

Hey there @Wanggcong

Thanks so much for making this repo available. It has been very interesting to explore!

I've been evaluating your pretrained model on some of the Laval dataset - I'm noticing in the paper you get quite close to ground truth results.

Whilst evaluating the model + code, I've been noticing that I'm struggling quite a lot to get close to ground truth. I've attempted various hyperparameter configurations. All the examples generated below were done with 600 steps on the projection step (generating the latent code) and 350 steps on the inversion (usually reaches the lpips threshold early, around 30-100 steps)

The crop2pano is set to 60 degrees for the fov and aspect ratio of the input is maintained but scaled down to fit in 480p

Here are some attached examples:

Input:

Ground truth:

Generated:
1.

I was wondering if you had any thoughts on what may be going wrong, or anything that can be modified to make the output more consistent with ground truth?

Many thanks in advance

Dear authors,

Thanks for your great work.

Sorry to bother you, I have a few questions.

1. After running test_lighting.py, I can get the panorama, but how can I get the light direction (and also intensity) of the input FOV image? In simple words, my need is, input a image, and I want to know the light direction of this image. For multiple light sources or without a main light source, is there a way to get the light direction and intensity per pixel level of the input FOV image?

2. What is the best size of the input FOV image for test_lighting.py, or the best ratio of width/height, for example width/height=2?

3. In terms of application on face relighting, how can I get faces like in Fig. 14 in your paper? I mean, is the input a cropped and aligned face? Is the same trained model that you offered on Google Drive? After running test_lighting.py, I get the panorama with a face in the center, and then what should I do to get faces like in Fig. 14?

Thanks in advance.

Dear authors,

Is there a way to “normalize” the light condition for a group of images?

For example, in face recognition, I have 1000 faces. For some faces, there is strong light from upper left, for some other faces, there is strong light from both upper left and right. For some faces, light is very strong, and for others light is very weak.

I want to get rid of the light effect, can I have an average or normalized light condition for all faces?

Thanks in advance.