Giter VIP home page Giter VIP logo

pixhtlab-src's Introduction

PixHtLab

This is the source code([Paper] [SSN Demo] [SSN Training] [PixHtLab Demo] [SSG & PixHtLab Training]) repository for three papers:

SSN

We released the training codes, a gradio demo (also hosted in huggingface) and the training dataset.

Dataset

The training dataset can created by code provided in the SSN old repo.

We provide a precomputed dataset composed of 558 different 3D models, which can be downloaded from our huggingface dataset repo. See the ssn_dataset.hdf5 file.

SSN-Demo

A gradio-based GUI demo is provided in the Demo/SSN folder. First, download the weight it used. Then run the following code to see the demo:

python app.py

SSN-Training

The training codes are under Train folder. To reproduce the training process, first prepare the dataset discussed above, then run the following command:

cd Train 

# before running, try to check the setting in SSN.yaml.
# The YAML file assumes the dataset file path is: Train/Dataset/SSN/ssn_dataset.hdf5
python app/Trainer.py --config configs/SSN.yaml

SSG & PixHtLab

We find shadow can be rendered using pixel height maps. We further explored rendering in pixel height representation and found not only shadow can be rendered, 3D light effects (shadow, reflection, refractions, etc) can be rendered as well. So SSG can be viewed as a subset of PixHtLab. For simplicity, this repo directly release the code for PixHtLab.

Environment Setup

First, create a conda environment with python 3.9.

conda create -n pixht python=3.9  -y
conda activate pixht

Then run the env.sh script. I noticed that the environment to setup the training framework is becoming to be a little tricky due to some python package updates. So I will build a Docker for the ease of use.

bash env.sh 

Dataset

As the file is large (173G), the dataset file is put in Dropbox. Use this link to download. The advantage of Dropbox link is that you can use wget to download.

# remember to cd to where you want to save the dataset file. 
wget -O dataset.hdf5 https://www.dropbox.com/scl/fi/ux7wr5uz2rne6vu70eq2f/dataset.hdf5?rlkey=pzubhj41m6j1muj393j33iuzm&dl=0

PixHtLab-Demo

For PixHtLab, we do not have Gradio demo as the computation relies on CUDA layer operation. The free Gradio demo on huggingface only provides CPU calculations. Instead, I put a jupyter notebook in the Demo/PixhtLab/Demo.ipynb. It provides an example to show how to use PixHtLab to render shadow and reflections.

Weight File

Soft shadow rendering needs pre-trained SSN. The weight can be downloaded from here. Create a weight folder (weights) under Demo/PixhtLab/ and put the weight there. Or use the following command to do this:

mkdir Demo/PixhtLab/weights
cd Demo/PixhtLab/weights

# wget to download 
wget -O human_baseline_all_21-July-04-52-AM.pt https://www.dropbox.com/scl/fi/7vzb0hlff0wbbb6l6qiyb/human_baseline_all_21-July-04-52-AM.pt?rlkey=lw34u1qao6had58t9vqy3bfbj&dl=0

This demo weight is a little old. But it should be enough for most cases for demonstration. If you want better weight, you can try to replace this weight with the latest SSN weight discussed above. I have not tested if directly replacing the weight is OK. If it is not OK, try to replace the inference module using the above SSN inference module.

PixHtLab-Training

The training codes are under Train folder. To reproduce the training process, first prepare the dataset discussed above. The dataset for SSG and SSN++ (sometimes I call it GSSN) is the same hdf5 file. Then run the following command:

cd Train 

# Before running, try to check the setting in SSG.yaml.
# The YAML file assumes the dataset file path is: Train/Dataset/SSG/dataset.hdf5
python app/Trainer.py --config configs/SSG.yaml

# or if you want to train SSN++ with 3D-aware buffer channels, try this:
# The YAML file assumes the dataset file path is: Train/Dataset/SSG/dataset.hdf5
python app/Trainer.py --config configs/GSSN.yaml

Updates

  • [2023-03-16] Basic setup.
  • [2024-01-02] SSN dataset/demo/inference/training
  • [2024-02-20] (Sry, I was busy at that time.) SSG/PixhtLab dataset/demo/inference/training
  • [2024-02-20] Python environment setup
  • Build a docker image

License

This code repo can only be used for non-commercial use only.

Citation

If you think the code/dataset is useful, please remember to cite the three papers:

@inproceedings{sheng2023pixht,
  title={PixHt-Lab: Pixel Height Based Light Effect Generation for Image Compositing},
  author={Sheng, Yichen and Zhang, Jianming and Philip, Julien and Hold-Geoffroy, Yannick and Sun, Xin and Zhang, He and Ling, Lu and Benes, Bedrich},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={16643--16653},
  year={2023}
}

@inproceedings{sheng2022controllable,
  title={Controllable shadow generation using pixel height maps},
  author={Sheng, Yichen and Liu, Yifan and Zhang, Jianming and Yin, Wei and Oztireli, A Cengiz and Zhang, He and Lin, Zhe and Shechtman, Eli and Benes, Bedrich},
  booktitle={European Conference on Computer Vision},
  pages={240--256},
  year={2022},
  organization={Springer}
}

@inproceedings{sheng2021ssn,
  title={SSN: Soft shadow network for image compositing},
  author={Sheng, Yichen and Zhang, Jianming and Benes, Bedrich},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={4380--4390},
  year={2021}
}

pixhtlab-src's People

Contributors

shengcn avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pixhtlab-src's Issues

demo

Your demo is inconsistent with the interface shown in the demonstration instructions. How can I use the demo?

Code release time estimate

First of all, thank you for publishing these two interesting papers about pixel height @ShengCN and all of the team !

I was wondering if you have an estimate on the moment the inference code/model will be made available? I would love to be able to use this in my research project without having to re-implement and re-train the model myself. If you don't think these will be made available in the near future, I would also like to know ๐Ÿ™‚, so I can plan accordingly.

I see that there is a Demo branch, but it doesn't seem to be documented yet and hasn't been merged.

Thank you and have a great day!

Pixel Height estimation dataset/weights

Hi Yichen,

thank you for your works and this repo. I find your soft shadow generation method based on estimated (predicted) height maps very interesting and I would really love to try it. I think that I found the model in the repo: PixHtLab-Src/Train/app/models/Sparse_PH.py. But I haven't found any weights, or configs, or training data for this height map estimator.

Is it possible for you to release this module for inference or for training?

Thank you!

No module named 'hshadow'

I want to run the PixHtLab-Demo, Demo/PixhtLab/Demo.ipynb. But when I run this jupyter file, it shows that 'No module named 'hshadow'', and I think 'hshadow' maybe a tool module that you composed. But after I checked the folder 'Demo/PixhtLab/', I haven't find the file named like 'hshadow.py'. My issue is whether this module is necessary and if it were necessary, can you supply this file and I'll be grateful to you.

pretrained weights

Hello,
Pretrained models are available or not?
model = inference_shadow.init_models('/home/ysheng/Documents/Research/GSSN/HardShadow/qtGUI/weights/human_baseline_all_21-July-04-52-AM.pt')

GSSN_model = SSN_Infernece('/home/ysheng/Documents/Research/GSSN/HardShadow/qtGUI/GSSN/weights/only_shadow/0000000200.pt')

No module named 'SSN_v1'

Hello, I'm trying to run demo inference by following Demo.ipynb file, but getting this error:

Traceback (most recent call last):
  File "/home/user/PixHtLab-Src/Demo/PixhtLab/demo.py", line 8, in <module>
    from hshadow_render import hshadow_render
  File "/home/user/PixHtLab-Src/Demo/PixhtLab/hshadow_render.py", line 12, in <module>
    from GSSN.inference_shadow import SSN_Infernece
  File "/home/user/PixHtLab-Src/Demo/PixhtLab/GSSN/inference_shadow.py", line 28, in <module>
    from SSN_v1 import SSN_v1
ModuleNotFoundError: No module named 'SSN_v1'

I found a hardcoded sys path in inference_shadow.py, can be this reason?

...
import sys
# sys.path.insert(0, '../../Training/app/models')
sys.path.insert(0, '/home/ysheng/Documents/Research/GSSN/Training/app/models')

from SSN_v1 import SSN_v1
from SSN import SSN
...
Full code (copied from Demo.ipynb)
from __future__ import print_function
from os.path import join 
from glob import glob 
from tqdm.auto import tqdm
from torchvision import utils
import matplotlib.pyplot as plt
import numpy as np
from hshadow_render import hshadow_render
from hshadow_render import style_hardshadow as softshadow_render

from camera import axis_camera
from reflect_render import *
from xyh_proj import xyz2xyh, xyh2xyz, compute_normal, get_ray_mat, normalize_vec3

def show(img, title=''):
    print(img.shape, img.min(), img.max())
    plt.figure(figsize=(15,10))
    plt.imshow(img)
    plt.title(title)
    plt.show()

def save_image(img, title='', filename='result.png'):
    print(img.shape, img.min(), img.max())  # Displaying image properties
    plt.figure(figsize=(15, 10))
    plt.imshow(img)  # Display the image
    plt.title(title)  # Set the title of the image
    plt.savefig(filename)  # Save the figure to a file
    plt.close()  # Close the figure to free up memory
    
def show_tensor(tensor,title=''):
    img = utils.make_grid(tensor).detach().cpu().numpy().transpose(1,2,0)
    show(img, title=title)

def parse_configs(config):
    with open(config, 'r') as stream:
        try:
            configs=yaml.safe_load(stream)
            return configs
        except yaml.YAMLError as exc:
            logging.error(exc)
            return {}

        
# shadow related functions 
def proj_horizon_plane(p):
    pp = p.copy()
    pp[1] = pp[1] + pp[2] 
    pp[2] = pp[2] * 0.0
    return pp

def compute_light_pos(p0, p1, light_horizon):
    """ Given a pair of xyh and light h, compute the light height
    """ 
    if len(p0) != 3 or len(p1) !=3:
        raise ValueError('Input size {},{} is wrong(should be 3).'.format(p0.shape, p1.shape))

    x0, y0, h0, x1, y1, h1 = p0[0], p0[1], p0[2], p1[0], p1[1], p1[2]
    light_sign = ((y1+h1)-(y0+h0))/np.abs((y1+h1)-(y0+h0))
    print('light sign: ', light_sign)

    # import pdb; pdb.set_trace()
    pp0, pp1 = proj_horizon_plane(p0), proj_horizon_plane(p1) 
    t = (light_horizon - pp0[1])/(pp1[1]-pp0[1])
    lightx, lighty, lighth = (1.0-t)*x0 + t*x1, (1.0-t)*y0+t*y1, (1.0-t)*h0 + t * h1
    return np.array([lightx, lighty, lighth])



def composite(rgb, mask, shadow):
    return rgb * mask + (1.0-mask) * shadow

def composite_reflection(rgb, mask, fg_pixht, bg_rgba, bg_pixht, rechmap, horizon, sample=10, glossness=0.05):
    params = {
        'sample_n': sample,
        'horizon': horizon,
        'ref_idx': 0.3,
        'glossness': glossness,
        'dh': 1.0,
        'batch_size': 1,
        'reflect_alpha': 1.0,
        'camera_h': 10
    }
    
    fg_rgba = np.concatenate([rgb, mask], axis=2)
    fg_height = fg_pixht
    bg_height = rechmap
    bg_reflection_layer = bg_rgba[..., -1:]

    reflection_rgb = reflection_layer(fg_rgba, fg_height, bg_rgba, bg_height, bg_reflection_layer, params)
    
    comp = rgb * mask + (1.0-mask) * reflection_rgb
    return comp


def render_shadow(rgb, mask, fg_pixht, bg_pixht, rechmap, x, y, horizon, softness=0.9):
    tmphmap = mask * fg_pixht
    mask_top_pos = list(np.unravel_index(np.argmax(tmphmap), tmphmap.shape))
    mask_top_pos = [mask_top_pos[1], mask_top_pos[0], tmphmap.max()]
    
    mouse_p = np.array([x, y, rechmap[y, x, 0]])
    light_xyh = compute_light_pos(mouse_p, mask_top_pos, horizon)
    
    tmp_mask = mask.copy()
    tmp_mask[tmp_mask>0.0] = 1.0
    
    fg_pixht = fg_pixht * mask
    bg_pixht = bg_pixht * mask
    
    fg_min = fg_pixht[tmp_mask != 0].min()
    fg_pixht -= fg_min
    bg_pixht -= fg_min
    
    last_pos = [0, 0]
    fg_shadow = hshadow_render(rgb, mask, fg_pixht, rechmap, light_xyh, last_pos)
    bg_shadow = hshadow_render(rgb, mask, bg_pixht, rechmap, light_xyh, last_pos)
    
    return fg_shadow, bg_shadow


def composite(rgb1, rgb2, mask1):
    return rgb1 * mask1 + rgb2 * (1.0-mask1)


def safe_comp(rgb1, rgb2, mask1, topleft):
    h1, w1 = rgb1.shape[:2]
    h2, w2 = rgb2.shape[:2]
    
    h_ = topleft[0]
    w_ = topleft[1]
    
    h__ = h_ + h1
    w__ = w_ + w1
    
    h_1 = np.clip(h_, 0, h2)
    w_1 = np.clip(w_, 0, w2)
    
    h__1 = np.clip(h__, 0, h2)
    w__1 = np.clip(w__, 0, w2)
    
    delta_h = h__ -  h__1
    delta_w = w__ - w__1
    
    cur_rgb1 = rgb1[:h1-delta_h, :w1 - delta_w]
    cur_mask1 = mask1[:h1-delta_h, :w1 - delta_w]
    
    rgb2[h_:h__1, w_:w__1] = composite(cur_rgb1, rgb2[h_:h__1, w_:w__1], cur_mask1)
    return rgb2


def main():
    root = 'Examples'
    fg_rgba1 = plt.imread(join(root, '009_rgb.png'))
    fg_rgba2 = plt.imread(join(root, '010_rgb.png'))
    fg_rgba3 = plt.imread(join(root, '011_rgb.png'))

    fg_pixht1 = np.load(join(root, '009_pixht_new.npy'))
    fg_pixht2 = np.load(join(root, '010_pixht_new.npy'))
    fg_pixht3 = np.load(join(root, '011_pixht_new.npy'))

    mask1 = np.load(join(root, '009_depth_valid_mask.npy'))
    mask2 = np.load(join(root, '010_depth_valid_mask.npy'))
    mask3 = np.load(join(root, '011_depth_valid_mask.npy'))

    bg = plt.imread(join(root, 'c82c09fc01e84282bc8870c263dcf81b_bg.jpg')) / 255.0

    cur_rgb = fg_rgba1.copy()
    cur_mask = mask1.copy()[..., None]
    cur_fg_pixht = fg_pixht1.copy()[..., None]
    cur_bg_pixht = fg_pixht1.copy()[..., None]

    h, w = bg.shape[:2]

    bg = cv2.resize(bg, (w//4, h//4))


    top_left1 = [500, 360]
    size1 = [512, 512]

    cur_slice_h = slice(top_left1[0], top_left1[0] + size1[0])
    cur_slice_w = slice(top_left1[1], top_left1[1] + size1[1])

    cur_bg = bg.copy()
    new_fg_rgb = np.zeros_like(cur_bg)
    new_fg_mask = np.zeros_like(cur_bg)
    new_fg_pixht = np.zeros_like(cur_bg)

    cur_bg = safe_comp(cur_rgb, cur_bg, cur_mask, top_left1)
    new_fg_rgb = safe_comp(cur_rgb, new_fg_rgb, cur_mask, top_left1)
    new_fg_mask = safe_comp(cur_mask, new_fg_mask, cur_mask, top_left1)

    new_fg_pixht = safe_comp(cur_fg_pixht, new_fg_pixht, cur_mask, top_left1)
    new_bg_pixht = new_fg_pixht.copy() 

    # show(new_fg_pixht)
    # show(cur_bg)
    # show(new_fg_rgb)
    # show(new_fg_mask)

    h, w = cur_bg.shape[:2]
    cur_bg_rgba = np.ones((h, w, 4))
    cur_bg_rgba[..., :3] = cur_bg


    rechmap = np.zeros_like(new_fg_rgb)[..., 0:1]

    new_fg_mask = new_fg_mask[..., 0:1]
    new_fg_pixht = new_fg_pixht[..., 0:1]
    new_bg_pixht = new_bg_pixht[..., 0:1]

    x = 455
    y = 776
    horizon = 600
    softness = 0.1

    print(new_fg_rgb.shape)
    print(new_fg_mask.shape)
    print(new_fg_pixht.shape)
    print(new_bg_pixht.shape)
    print(rechmap.shape)

    ao_shadow, bg_shadow = render_shadow(new_fg_rgb, new_fg_mask, new_fg_pixht, new_bg_pixht, rechmap, x, y, horizon)
    fg_shadow, bg_shadow = render_shadow(new_fg_rgb, new_fg_mask, new_fg_pixht, new_bg_pixht, rechmap, 455, 776, horizon)

    ao_weight = 0.5
    ao_softness = 0.2
    glossness = 0.01
    sample = 1

    ao_ss = softshadow_render(new_fg_mask[..., :1], ao_shadow[..., :1], ao_softness)[0]
    dir_ss = softshadow_render(new_fg_mask[..., :1], fg_shadow[..., :1], softness)[0]
    final_ss = 1.0 - ((1.0 - ao_ss) * ao_weight  + (1.0 - dir_ss) * (1.0-ao_weight) )
    shadow_bg = composite(new_fg_rgb, final_ss * cur_bg_rgba[..., :3], new_fg_mask)
    cur_bg_rgba[..., :3] = shadow_bg
    final = composite_reflection(new_fg_rgb, new_fg_mask, new_fg_pixht, cur_bg_rgba, new_bg_pixht, rechmap, horizon=horizon, glossness=glossness, sample=sample) 

    show(final)
    
if __name__ == '__main__':
    save_image()

Machine/OS Details:
Debian 10
CUDA 11.3
Python 3.9 [miniconda]
I installed everything by following env.sh file

dependent of imagen library

Nice work, but when i run training of ssn, some errors always happen because of "imagen",could you offer complete library of imagen or give some instruction about it? I will realy appreciate your help

A question about IBL in gradio.

Hello, thank you for sharing your valuable research.
What does IBL refer to in the SSN paper?
Are you referring to AO maps?
If it's a Light Map, do you use the inverse of 0 and 255 for the model input?
Thanks

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.