- Note: for Azure ML support for this repository, please consider checking out this branch by Stan Szymanowicz.
This code is developed with Python3 (python3
). PyTorch 1.9+ is required.
It is recommended use Anaconda to set up the environment. Install the dependencies and activate the environment barf-env
with
conda env create --file requirements.yaml python=3
conda activate barf-env
Initialize the external submodule dependencies with
git submodule update --init --recursive
- Acquire data using Strayscanner.
To generate the train/test data:
python data/process_strayscanner_data_image_resize.py --basedir <your_scenedir>
python data/process_strayscanner_data_image_resize.py --basedir ./data/strayscanner/chair
```f_llff_data data/llff
The data
directory should contain the subdirectories strayscanner
and iphone
.
If you already have the datasets downloaded, you can alternatively soft-link them within the data
directory.
-
Place your scene directory somewhere. Follow the Strayscanner data structure.
If you want to try CMNeRF on your own sequence, we provide a template data file in
data/strayscanner.py
, which is an example to read from a sequence captured by an iPhone 12 pro. You should modifyget_image()
to read each image sample and set the raw image sizes (self.raw_H
,self.raw_W
) and focal length (self.focal
) and (self.pose
) w2c pose([right,down,forward]) and depth information (self.gt_depth
,self.confidence
) according to your camera specs.
Note: if you use this data format, make sure yourdataset_type
in the config file is set asstrayscanner
.
-
To train and evaluate CMNeRF:
# <GROUP> and <NAME> can be set to your likes, while <SCENE> is specific to datasets #strayscanner python3 train.py --group=strayscanner --model=barf --yaml=barf_strayscanner --name=result_chair --data.scene=chair --barf_c2f=[0.1,0.5] --depth.use_depth=true --depth.use_depth_loss=true python3 evaluate.py --group=strayscanner --model=barf --yaml=barf_strayscanner --name=result_chair --data.scene=chair --data.val_sub= --depth.use_depth=true --depth.use_depth_loss=true #iphone (If you want to train the reference BARF models) python3 train.py --group=iphone --model=barf --yaml=barf_iphone --name=result_chair --data.scene=chair --barf_c2f=[0.1,0.5] --depth.use_depth=false --depth.use_depth_loss=false python3 evaluate.py --group=iphone --model=barf --yaml=barf_iphone --name=result_chair --data.scene=chair --data.val_sub= --resume --depth.use_depth=false --depth.use_depth_loss=false
All the results will be stored in the directory
output/<GROUP>/<NAME>
. You may want to organize your experiments by grouping different runs in the same group.To train baseline models:
- Full positional encoding: omit the
--barf_c2f
argument. - No positional encoding: add
--arch.posenc!
. --group=iphone
is a group that that does not use pose information in the data of the strayscanner.- If you want to evaluate a checkpoint at a specific iteration number, use
--resume=<ITER_NUMBER>
instead of just--resume
. - iphone 그룹은 초기 포즈를 identity로 설정한 것으로 strayscanner 그룹과 같은 데이터를 사용한다.
A video
vis.mp4
will also be created to visualize the optimization process. - Full positional encoding: omit the
The main engine and network architecture in model/barf.py
inherit those from model/nerf.py
.
This codebase is structured so that it is easy to understand the actual parts BARF is extending from NeRF.
(This code cannot be used to train origin NeRF.)
Some tips on using and understanding the codebase:
- The computation graph for forward/backprop is stored in
var
throughout the codebase. - The losses are stored in
loss
. To add a new loss function, just implement it incompute_loss()
and add its weight toopt.loss_weight.<name>
. It will automatically be added to the overall loss and logged to Tensorboard. - If you are using a multi-GPU machine, you can add
--gpu=<gpu_number>
to specify which GPU to use. Multi-GPU training/evaluation is currently not supported. - To resume from a previous checkpoint, add
--resume=<ITER_NUMBER>
, or just--resume
to resume from the latest checkpoint. - (to be continued....)
If you find our code useful for your research, please cite
@inproceedings{lin2021barf,
title={BARF: Bundle-Adjusting Neural Radiance Fields},
author={Lin, Chen-Hsuan and Ma, Wei-Chiu and Torralba, Antonio and Lucey, Simon},
booktitle={IEEE International Conference on Computer Vision ({ICCV})},
year={2021}
}
@inproceedings{roessle2022dense,
title={Dense depth priors for neural radiance fields from sparse input views},
author={Roessle, Barbara and Barron, Jonathan T and Mildenhall, Ben and Srinivasan, Pratul P and Nie{\ss}ner, Matthias},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={12892--12901},
year={2022}
}
MINE!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
This code borrows heavily from BARF. We thank BARF, from which this repository borrows code.
Please contact me ([email protected]) if you have any questions!