High-Dimensional Dense Residual Convolutional Neural Network for Light Field Reconstruction

This Project is a Tensorflow implementation of

"High-Dimensional Dense Residual Convolutional Neural Network for Light Field Reconstruction" IEEE Transactions on Pattern Analysis and Machine Intelligence, Nan Meng, Hayden K-H. So, Xing Sun, Edmund Y. Lam, 2019. [Paper]
"High-order Residual Network for Light Field Super-Resolution" The 34th AAAI Conference on Artificial Intelligence, Nan Meng, Xiaofei Wu, Jianzhuang Liu, Edmund Y. Lam, 2020. [Paper]

Requirements

Python2==2.7
Python3>=3.5
Tensorflow r1.*>=r1.8
tqmd
opencv
unrar

Installation

Download the project

git clone https://github.com/monaen/LightFieldReconstruction.git --branch master --single-branch

Training

Training models for spatial or angular or both super-resolution tasks

Training the model for spatial super-resolution (e.g. Sx4). You need to specify the gamma_S for different spatial SR tasks.

python train_SpatialSR.py --datadir data/train/Spatial/5x5
                          --gamma_S 4
                          --verbose
                          --perceptual_loss

Training the model for angular super-resolution (e.g. Ax4). You need to specify the gamma_A for different angular SR tasks.

python train_ViewSysthesis.py --datadir data/train/Angular/9x9
                              --gamma_A 4
                              --verbose

Training the model for spatio-angular super-resolution (e.g. Sx2Ax2). You need to specify both the gamma_S and gamma_A for different spatio-angular SR tasks.

python train_SpatialAngularSR.py --datadir data/train/SpatialAngular/5x5
                                 --gamma_S 2
                                 --gamma_A 2
                                 --verbose
                                 --perceptual_loss

Training arguments

usage:train_SpatialSR.py [-h] [--datadir DATADIR] [--lr_start LR_START] [--lr_beta1 LR_BETA1] [--batchSize BATCHSIZE]
                         [--imageSize IMAGESIZE] [--viewSize VIEWSIZE] [--channels CHANNELS] [--verbose VERBOSE]
                         [--num_epoch NUM_EPOCH] [--start_epoch START_EPOCH] [--gamma_S {1,2,3,4}]
                         [--gamma_A {0,1,2,3,4}] [--num_GRL_HRB NUM_GRL_HRB] [--num_SRe_HRB NUM_SRE_HRB]
                         [--resume RESUME] [--select_gpu SELECT_GPU] [--perceptual_loss PERCEPTUAL_LOSS]
                         [--vgg_model VGG_MODEL] [--save_folder SAVE_FOLDER]

optional arguments:
  -h, --help                Show this help message and exit
  --datadir                 The training and testing data path
  --lr_start                The start learning rate
  --lr_beta1                The exponential decay rate for the 1st moment estimates
  --batchSize               The batchsize of the input data
  --imageSize               Spatial size of the input light fields
  --viewSize                Angular size of the input light fields
  --channels                Channels=1 means only the luma channel; Channels=3 means RGB channels (not supported)
  --verbose                 Whether print the network structure or not
  --num_epoch               The total number of training epoch
  --start_epoch             The start epoch counting number
  --gamma_S {1,2,3,4}       Spatial downscaling factor
  --gamma_A {0,1,2,3,4}     Angular downscaling factor, '0' represents 3x3->7x7
  --num_GRL_HRB             The number of HRB in GRLNet (only for AAAI model)
  --num_SRe_HRB             The number of HRB in SReNet (only for AAAI model)
  --resume                  Need to resume the pretrained model or not
  --select_gpu              Select the gpu for training or evaluation
  --perceptual_loss         Need to use perceptual loss or not, if true, one also have to set the vgg_model item
  --vgg_model               Pretrained VGG model path
  --save_folder             Model save path

Training data samples

We provide the training samples to facilitate users to generate custom training set.

# change to the root folder 'LightFieldReconstruction' of the project
cd data
bash download_trainingsamples.sh

Evaluation

Spatial SR evaluation (Sx4, Sx3, Sx2)

python evaluation_SpatialSR.py --datapath $PATH_TO_THE_EVALUATION_LF_DATA \
                               --gamma_S $Upscaling_Factor \
                               --pretrained_model $PATH_TO_THE_PRETRAINED_MODEL \
                               --select_gpu 0 --verbose

Here, we provide an example to evaluate the performance on Sx4 task as guidance.

# change to the root folder 'LightFieldReconstruction' of the project
cd data
bash download_evaluation_data.sh
cd ../pretrained_models
bash download_pretrained_models_HDDRNet_Sx4.sh
cd ..

python evaluation_SpatialSR.py --datapath data/evaluation/buddha.mat \
                               --gamma_S 4 \
                               --pretrained_model pretrained_models/HDDRNet/Sx4/HDDRNet \
                               --select_gpu 0 --verbose

Angular SR evaluation (Ax4, Ax3, Ax2, A3x3_7x7)

# change to the root folder 'LightFieldReconstruction' of the project
cd data
bash download_occlusions20.sh
cd ../pretrained_models
bash download_pretrained_models_HDDRNet_Ax4.sh
cd ..

python evaluation_ViewSynthesis.py --datapath data/testset/occlusions20/occlusions_48.mat \
                                   --gamma_A 4 \
                                   --pretrained_model pretrained_models/HDDRNet/Ax4/HDDRNet \
                                   --select_gpu 0 --verbose


python evaluation_ViewSynthesis_Patchwise.py --datapath data/testset/occlusions20/occlusions_48.mat \
                                             --gamma_A 4 \
                                             --pretrained_model pretrained_models/HDDRNet/Ax4/HDDRNet \
                                             --select_gpu 0 --verbose

Spatial and Angular SR evaluation (Sx2Ax2, Sx3Ax2)

Note: We provide the pretrained model for Sx2Ax2, but the model is not well-trained. We also provide the code for Sx3Ax2 task. Here, we provide an example to evaluate the performance on Sx2Ax2 task as guidance.

# change to the root folder 'LightFieldReconstruction' of the project
cd data
bash download_evaluation_data.sh
cd ../pretrained_models
bash download_pretrained_models_Sx2Ax2.sh
cd ..

python evaluation_SpatialAngularSR.py --datapath data/evaluation/buddha.mat \
                                      --gamma_S 2 \
                                      --gamma_A 2 \
                                      --pretrained_model pretrained_models/Others/Sx2Ax2/weights \
                                      --select_gpu 0 --verbose

Evaluate the entire dataset

Note: All the reported results are tested using Matlab, and here we provide an example to show how to evaluate the results on reflective20 dataset.

# change to the root folder 'LightFieldReconstruction' of the project
cd data
bash download_reflective20.sh
cd ../pretrained_models
bash download_pretrained_models_M-HDDRNet_Ax4.sh
cd ..

python evaluation_on_dataset.py --datafolder data/testset/reflective20 \
                                --gamma_S 1 \
                                --gamma_A 4 \
                                --pretrained_model pretrained_models/M-HDDRNet/Ax4/M-HDDRNet \
                                --result_folder results \
                                --select_gpu 0 --verbose


# After the execution of "evaluation_on_dataset.py", run the m-file "evaluation.m"

matlab -nodisplay -nosplash -nodesktop -r "run evaluation.m; exit;"

Using the pretrained model

To download the pretrained mdoels, please change the directory into the folder pretrained_models and run the corresponding bash files. For example, to download the HDDRNet_Sx4 pretrained model,

# path = Path to LightFieldReconstruction
cd pretrained_models
bash download_pretrained_models_HDDRNet_Sx4.sh

We provide a detailed instruction on how to download the pretrained models for differnet SR models and tasks.

Reference

Paper

@article{Meng2019High,
  title        = {High-dimensional dense residual convolutional neural network for light field reconstruction},
  author       = {Meng, Nan and So, Hayden Kwok-Hay and Sun, Xing and Lam, Edmund},
  journal      = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year         = {2019}
}
@article{Meng2020High,
  title        = {High-order residual network for light field super-resolution},
  author       = {Meng, Nan and Wu, Xiaofei and Liu, Jianzhuang and Lam, Edmund},
  journal      = {Association for the Advancement of Artificial Intelligence},
  volume       = {34},
  number       = {7},
  pages        = {11757-11764},
  month        = {February},
  year         = {2020}
}

Frequently Asked Questions

TODO List

T-PAMI

AAAI

Spatial super-resolution 2x framework.
Spatial super-resolution 3x framework.
Spatial super-resolution 4x framework.
Angular super-resolution 2x (5x5->9x9).
Angular super-resolution 2.3x (3x3->7x7).
Angular super-resolution 3x (3x3->9x9).
Angular super-resolution 4x (2x2->8x8).
Other materials.

Matlab

Matlab evaluation code.

For some reasons, we are not able to provide the pre-trained models for the AAAI work.

Acknowledgement

Our code is released under MIT License. We would like to appreciate the GPU support from Lee Ultrasound Imaging Group of Prof.Wei-Ning Lee

monaen / lightfieldreconstruction Goto Github PK

lightfieldreconstruction's Introduction

High-Dimensional Dense Residual Convolutional Neural Network for Light Field Reconstruction

Requirements

Installation

Training

Training models for spatial or angular or both super-resolution tasks

Training arguments

Training data samples

Evaluation

Using the pretrained model

Reference

Frequently Asked Questions

TODO List

Acknowledgement

lightfieldreconstruction's People

Contributors

Stargazers

Watchers

Forkers

lightfieldreconstruction's Issues

Recommend Projects

Recommend Topics

Recommend Org