The official implementation of "Bayesian Image Super-Resolution with Deep Modeling of Image Statistics" which has been accepted by IEEE Transactions on Pattern Analysis and Machine Intelligence.

• BayeSR
  • Content
  • Wiki
  • Dependencies
  • Quick test
    • Ideal image super-resolution
    • Realistic image super-resolution
    • Real-world image super-resolution
    • Synthetic image super-resolution
  • How to train
    • Prepare training datasets
    • Start training
  • Citation

Please refer to wiki page for the detailed introduction of BayeSR.

BayeSR was implemented on Ubuntu 16.04 with Python 3.6. Before training and test, please create an environment via Anaconda (suppose it has been installed on your computer), and install tensorflow 1.10.0, as follows,

conda create -n BayeSR python=3.6
source activate BayeSR
conda install tensorflow-gpu==1.10.0

Besides, please install the following packages using pip install -r requirements.txt.

• numpy==1.14.5
• opencv-python==4.4.0
• tqdm==4.51.0
• scikit-image==0.17.1
• Pillow==8.0.1
• scipy==1.2.1

BayeSR was tested on the widely used benchmark datasets for the tasks of ideal SISR x4, realistic SISR x4, and real-world SISR x4. We have provided the benchmark datasets and the pre-trained models via the following links and extraction codes.

• benchmark.zip contains seven datasets:
  • Set5, Set14, B100, and Urban100 for ideal SISR
  • DIV2K_mild for realistic SISR
  • DPED_iphone for real-world SISR
  • RealWorld for test in real-world scenarios.
• downsamplors.zip contains four downsampling models:
  • BayeSR-Downsamplor-Bicubicx4 trained by supervised learning for ideal SISR x4
  • BayeSR-Downsamplor-BiSRx4 trained by unsupervised KernelGAN for ideal SISR x4
  • BayeSR-Downsamplor-ReSRx4 trained by unsupervised KernelGAN for realistic SISR x4
  • BayeSR-Downsamplor-RWSRx4 trained by unsupervised KernelGAN for real-world SISR x4
• IdealSISR.zip contains two BayeSR models for ideal SISR x4:
  • BayeSR-BiSRx4-sup trained by the supervised strategy of BayeSR
  • BayeSR-BiSRx4-unsup trained by the unsupervised strategy of BayeSR
• RealisticSISR.zip contains three BayeSR models for realistic SISR x4:
  • BayeSR-ReSRx4-sup trained by the supervised strategy of BayeSR
  • BayeSR-ReSRx4-psup trained by the pseudo-supervised strategy of BayeSR
  • BayeSR-ReSRx4-unsup trained by the unsupervised strategy of BayeSR
• RealworldSISR.zip contains two BayeSR models for real-world SISR x4:
  • BayeSR-RWSRx4-psup trained by the pseudo-supervised strategy of BayeSR
  • BayeSR-RWSRx4-unsup trained by the unsupervised strategy of BayeSR
• SyntheticSISR.zip contains one BayeSR models:
  • BayeSR-SySRx4-sup trained by the supervised strategy of BayeSR, where LR images were corrupted by stochastic Gaussian kernel and Gaussian noise, instead of a pret-trained downsampling model.

We have provided the script of testing BayeSR in src/demo.sh. Please cd your working path to src/ and start test BayeSR as follows.

To test the generalizability of BayeSR on ideal SISR x4, please uncomment the following line in demo.sh, and then run sh demo.sh.

# supervised learning
python bayesr_test.py --dataset Set5 --input_data_dir ../data/TestDatasets/benchmark --task BiSR --sigma_read 20 --bayesr_checkpoint ../models/IdealSISR/BayeSR-BiSRx4-sup/model --GPU_ids 0

Here, --sigma_read defines the level of addictive white Gaussian noise. Note that the printed PSNR and SSIM values may be higher than the reported PSNR and SSIM in our article, since the latter were computed by a MATLAB script (utils/PSNR_SSIM/PSNR_SSIM.m) consistent with the previous works such as SRCNN and VDSR.

To test the unsupervised performance of BayeSR on ideal SISR x4, please uncomment the following line in demo.sh, and then run sh demo.sh.

# unsupervised learning
python bayesr_test.py --dataset Set5 --input_data_dir ../data/TestDatasets/benchmark --task BiSR --bayesr_checkpoint ../models/IdealSISR/BayeSR-BiSRx4-unsup/model --GPU_ids 0

To test the supervised, pseudo-supervised, and unsupervised performance of BayeSR on realistic SISR x4, please uncomment one of the following lines in demo.sh, and then run sh demo.sh.

# supervised learning
python bayesr_test.py --dataset DIV2K_mild --input_data_dir ../data/TestDatasets/benchmark --task ReSR --bayesr_checkpoint ../models/RealisticSISR/BayeSR-ReSRx4-sup/model --GPU_ids 0
# pseudo-supervised learning
python bayesr_test.py --dataset DIV2K_mild --input_data_dir ../data/TestDatasets/benchmark --task ReSR --bayesr_checkpoint ../models/RealisticSISR/BayeSR-ReSRx4-psup/model --GPU_ids 0
# unsupervised learning
python bayesr_test.py --dataset DIV2K_mild --input_data_dir ../data/TestDatasets/benchmark --task ReSR --bayesr_checkpoint ../models/RealisticSISR/BayeSR-ReSRx4-unsup/model --GPU_ids 0

To test the pseudo-supervised and unsupervised performance of BayeSR, please uncomment one of the following lines in demo.sh, and then run sh demo.sh.

# pseudo-supervised learning
python bayesr_test.py --dataset DPED_iphone --input_data_dir ../data/TestDatasets/benchmark --task RWSR --bayesr_checkpoint ../models/RealworldSISR/BayeSR-RWSRx4-psup/model --GPU_ids 0
# unsupervised learning
python bayesr_test.py --dataset DPED_iphone --input_data_dir ../data/TestDatasets/benchmark --task RWSR --bayesr_checkpoint ../models/RealworldSISR/BayeSR-RWSRx4-unsup/model --GPU_ids 0

To test the generalizability of BayeSR to diverse degradation kernels, please uncomment the following line in demo.sh, and then run sh demo.sh.

# supervised learning
python bayesr_kernels.py --dataset Set14 --input_data_dir ../data/TestDatasets/benchmark --task SySR --bayesr_checkpoint ../models/SyntheticSISR/BayeSR-SySRx4-sup/model --GPU_ids 0

Here, the default degradation kerenl is the kernel with respect to bicubic interpolation.

To test the performance of BayeSR in real-world scenarios, please uncomment the following line in demo.sh, and then run sh demo.sh.

# supervised learning
python bayesr_test.py --dataset RealWorld --input_data_dir ../data/TestDatasets/benchmark --task RWSR --bayesr_checkpoint ../models/SyntheticSISR/BayeSR-SySRx4-sup/model --GPU_ids 0

Here, the dataset RealWorld can be replaced with your own real-world image dataset.

Before training, please prepare training datasets at first. Then, the downsampling module of BayeSR was pretrained by the unsupervised KernelGAN. Finally, BayeSR was trained by fixing the downsampling model.

DIV2K, Flickr2K, and DPED were used to train BayeSR. All datasets should be included in the folder data/TrainDatasets/ following a specific directory structure as follows.

DIV2K
  |--DIV2K_HR
       |--DIV2K_HR_train
       |--DIV2K_HR_valid
  |--DIV2K_LR_bicubic
       |--DIV2K_LR_train
            |--X4
       |--DIV2K_LR_valid
            |--X4
  |--DIV2K_LR_mild
       |--DIV2K_LR_train
            |--X4
            |--X4_noise
Flickr2K
  |--Flickr2K_HR
DPED
  |--iphone
       |--train_LR
       |--train_LR_noise
       |--valid_LR

Here, *_noise contains extracted noise patches from the LR image dataset *. For examples, DPED/iphone/train_LR_noise can be generated by cd to data/ and running python extract_nosie.py.

To pre-train downsampling models, please uncomment the following line in demo.sh, and run sh demo.sh.

# unsupervised learning
python bayesr_train.py --task RWSR --trained_net Downsamplor --train_type unsupervised --log_dir ../models/downsamplors/BayeSR-Downsamplor-RWSRx4 --GPU_ids 0

Here, --task can be set to BiSR, ReSR, and RWSR. The resulting downsampling models are corresponding to ideal SISR, realistic SISR, and real-world SISR.

To train BayeSR on the task of ideal SISR x4, please uncomment one of the following line in demo.sh, and run sh demo.sh.

# supervised learning
python bayesr_train.py --task BiSR --train_type supervised --downsamplor_checkpoint ../models/downsamplors/BayeSR-Downsamplor-BiSRx4/model --log_dir ../models/IdealSISR/BayeSR-BiSRx4-sup --GPU_ids 0
# unsupervised learning
python bayesr_train.py --task BiSR --GAN_type GAN --train_type unsupervised --downsamplor_checkpoint ../models/downsamplors/BayeSR-Downsamplor-BiSRx4/model --log_dir ../models/IdealSISR/BayeSR-BiSRx4-unsup --GPU_ids 0

Here, --downsamplor_checkpoint points to the path of the pre-trained downsampling model for ideal SISR x4. --GAN_type defines the types of generative adversarial networks (GANs), and the choices are GAN, WGAN, and LSGAN.

To train BayeSR on the task of realistic SISR x4, please uncomment one of the following lines in demo.sh, and run sh demo.sh.

# supervised learning
python bayesr_train.py --task ReSR --train_type supervised --downsamplor_checkpoint ../models/downsamplors/BayeSR-Downsamplor-ReSRx4/model --log_dir ../models/RealisticSISR/BayeSR-ReSRx4-sup --GPU_ids 0
# pseudo-supervised learning
python bayesr_train.py --task ReSR --GAN_type GAN --train_type pseudosupervised --downsamplor_checkpoint ../models/downsamplors/BayeSR-Downsamplor-ReSRx4/model --log_dir ../models/RealisticSISR/BayeSR-ReSRx4-psup --GPU_ids 0
# unsupervised learning
python bayesr_train.py --task ReSR --GAN_type GAN --train_type unsupervised --downsamplor_checkpoint ../models/downsamplors/BayeSR-Downsamplor-ReSRx4/model --log_dir ../models/RealisticSISR/BayeSR-ReSRx4-unsup --GPU_ids 0

To train BayeSR on the task of real-world SISR x4, please uncommnet one of the following lines in demo.sh, and then run sh demo.sh.

# pseudo-supervised learning
python bayesr_train.py --task RWSR --GAN_type GAN --train_type pseudosupervised --downsamplor_checkpoint ../models/downsamplors/BayeSR-Downsamplor-RWSRx4/model --log_dir ../models/RealworldSISR/BayeSR-RWSRx4-psup --GPU_ids 0
# unsupervised learning
python bayesr_train.py --task RWSR --GAN_type GAN --train_type unsupervised --downsamplor_checkpoint ../models/downsamplors/BayeSR-Downsamplor-RWSRx4/model --log_dir ../models/RealworldSISR/BayeSR-RWSRx4-unsup --GPU_ids 0

To train BayeSR on the task of synthetic SISR x4, where LR images were degraded by random Gaussian kernel and Gaussian noise, please uncomment the following line in demo.sh, and then run sh demo.sh.

# supervised learning
python bayesr_train.py --task SySR --train_type supervised --gauss_blur --downsamplor_checkpoint ../models/downsamplors/BayeSR-Downsamplor-BiSRx4/model --log_dir ../models/SyntheticSISR/BayeSR-SySRx4-sup --GPU_ids 0

If our work is helpful in your research, please cite this as follows.

[1] S. Gao and X. Zhuang, "Bayesian Image Super-Resolution with Deep Modeling of Image Statistics," in IEEE Transactions on Pattern Analysis and Machine Intelligence, doi: 10.1109/TPAMI.2022.3163307. [arXiv] [TPAMI]

@ARTICLE{9744488,
  author={Gao, Shangqi and Zhuang, Xiahai},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, 
  title={Bayesian Image Super-Resolution with Deep Modeling of Image Statistics}, 
  year={2022},
  volume={},
  number={},
  pages={1-1},
  doi={10.1109/TPAMI.2022.3163307}}

Don't hesitate to contact us via [email protected] or [email protected], if you have any questions.