aibluefisher / dreg-nerf Goto Github PK

Official Implementation of our ICCV 2023 paper: "DReg-NeRF: Deep Registration for Neural Radiance Fields"

Home Page: https://aibluefisher.github.io/DReg-NeRF/

Python 97.78% Shell 2.22%

3d-computer-vision computer-vision nerf registration

dreg-nerf's Introduction

DReg-NeRF

Official Implementation of our ICCV 2023 paper: "DReg-NeRF: Deep Registration for Neural Radiance Fields". [Project Page | ArXiv)

Given a pair of NeRF models that are trained in different coordinate frames, DReg-NeRF registers them into the same coordinate frame

1. Installation

conda create -n dreg_nerf python=3.9
conda activate dreg_nerf
cd DReg-NeRF/scripts
./scripts/env/install.sh

After installing the environment, you can try our pretrained model by following # 4.2 if you don't want to re-train all the NeRF models from scratch.

2. Fetch 3D Models from Objaverse

# Firstly, install conda env for blender:
cd DReg-NeRF/scripts/blender
pip install -r blender_environment.yml
# Then, we download models from Objaverse and render images.
./objaverse_to_blender.sh

After rendering images from 3D meshes, we put all the rendered images into images folder.

3. Train NeRF Blocks

Suppose the rendered images are put into ${DATA_ROOT_DIR}/objaverse/images.

3.1 Train

cd scripts/train
GPU_IDX=0
./train_objaverse_batch.sh true ${GPU_IDX}

After training, checkpoints/models are stored at ${DATA_ROOT_DIR}/objaverse/images/out.

3.2 Evaluation & Preprocessing

cd scripts/eval
GPU_IDX=0
./eval_objaverse_batch.sh true ${GPU_IDX}

After evaluation, results are stored at ${DATA_ROOT_DIR}/objaverse/images/eval. During evaluation, we also pre-extracted occupancy grids from the NeRF models in each block and store them at ${DATA_ROOT_DIR}/objaverse/images/out. To save disk storage, we only store model.pth, voxel_grid.pt, voxel_mask.pt and voxel_point_cloud.ply in each NeRF block models. Before training DReg-NeRF, you should check whether the trained NeRF models are good enough.

4. Train DReg-NeRF

Before training DReg-NeRF, we put the pre-extracted occupancy grids and NeRF models into a folder named nerf_models and put in at the same level to the images folder. Then, all the input required to train DReg-NeRF are stored at ${DATA_ROOT_DIR}/objaverse/nerf_models.

4.1 Train

cd scripts/train
GPU_IDX=0
./train_nerf_regtr.sh ${GPU_IDX}

After training, checkpoints are stored at ${DATA_ROOT_DIR}/out/$expname.

4.2 Evaluation

Downloading the evaluation data to test DReg-NeRF.

cd scripts/train
GPU_IDX=0
./eval_nerf_regtr.sh objaverse test 0

After evaluation, results are stored at ~/DATASETS/dreg_nerf/eval/$expname.

Cite

If you find our work useful for your research, please cite our paper as below:

@article{DBLP:journals/corr/abs-2308-09386,
  author    = {Yu Chen and
               Gim Hee Lee},
  title     = {DReg-NeRF: Deep Registration for Neural Radiance Fields},
  booktitle = {2023 {IEEE/CVF} International Conference on Computer Vision},
  pages     = {},
  year      = {2023},
}

dreg-nerf's People

Contributors

Stargazers

Watchers

Forkers

whuhxb

dreg-nerf's Issues

Runtime Error

Great work! an error occurred. details as follow：

2 Section: Fetch 3D Models from Objaverse,
when bash objaverse_to_blender.sh / print(selected_uids)
occurred bugs appear :
['S', 't', 'i', 'l', 'l', '', 'L', 'i', 'f', 'e', '', '_', 'L', 'e', 'm', 'o', 'n'...........
It seems wrong to do this, but I'm going to change the code like this

def load_obj_uids(json_file):
    selected_uids = []
    with open(json_file, 'r') as f:
        data = json.load(f)
    
    for key, item in data.items():
        #  selected_uids += item
            selected_uids += keys
    
    return selected_uids

def load_obj_uids(json_file):
    selected_uids = []
    with open(json_file, 'r') as f:
        data = json.load(f)
    
    for key, item in data.items():
        uid = key.split('/')[0]
        selected_uids.append(uid)
    
    return selected_uids

Its Okay! But internet problem
urllib.error.URLError: <urlopen error [Errno -3] Temporary failure in name resolution>
Then try to hf-mirror.com replace huggface.co,also some thing happen
File

"/home/shuaige/anaconda3/envs/blender/lib/python3.10/urllib/request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 401: Unauthorized

Cannot install bpy using python 3.9

While I was trying to install bpy, I run the script with

conda env update --file blender_environment.yml

Which returned:

ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: 'DReg-NeRF/scripts/blender/bpy-3.4.0a0-cp310-cp310-manylinux_2_31_x86_64.whl'

And when I tried to install with

pip install bpy

It said:

ERROR: No matching distribution found for bpy

error when running inference

Thanks for your code. When I ran your code, I met the problem with the tinycudnn as follows,

File "/workspace/project/DReg-NeRF/conerf/utils/utils.py", line 86, in sigma_fn
return radiance_field.query_density(positions)
File "/workspace/project/DReg-NeRF/conerf/radiance_fields/ngp.py", line 157, in query_density
self.mlp_base(x.view(-1, self.num_dim))
File "/root/miniconda3/envs/nerf/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/root/miniconda3/envs/nerf/lib/python3.9/site-packages/tinycudann/modules.py", line 186, in forward
output = _module_function.apply(
File "/root/miniconda3/envs/nerf/lib/python3.9/site-packages/tinycudann/modules.py", line 98, in forward
native_ctx, output = native_tcnn_module.fwd(input, params)
RuntimeError: /tmp/pip-req-build-3a45ith0/include/tiny-cuda-nn/cutlass_matmul.h:330 status failed with error Error Internal

I can use tinycudnn in Nerfstudio and instantNGP successfully. I only met this problem when running your code. What do you think the problem is? I am looking forward to your answer.

Performance of DReg-NeRG on low- or zero overlapping objects

Hi,

Congrats for your work and thanks for making it available.

As the title of the issues points out, I would like to ask whether you have done any experiments or tested how your solution performs on low- or zero overlapping objects.

For example imagine that you have two individual pieces of a complete object (broken object in two pieces), and for each you create a NeRF model as you do in your case scenario and you would like to apply an assembly task (i.e. registration with low- or no-overlap). Would your approach work in such a case scenario or would be possible to be modified for such a case?

Thanks.

RuntimeError: Trying to create tensor with negative dimension -1929448213: [-1929448213, 1]

Thanks for your code. When I ran your code, I met the problem：RuntimeError: Trying to create tensor with negative dimension -1929448213: [-1929448213, 1] as follows,
Evaluating train_nerf_regtr_obj NeRFRegTR: 0%| | 0/1 [00:00<?, ?it/s]pose_error: {'R_error_mean': tensor(124.2926), 't_error_mean': tensor(0.2078, device='cuda:0'), 'R_error_med': tensor(124.2926), 't_error_med': tensor(0.2078, device='cuda:0')} data batch: 4b05 /data/ldzhang/Projects/DReg-NeRF/eval_nerf_regtr.py:306: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /opt/conda/conda-bld/pytorch_1666642975993/work/torch/csrc/utils/tensor_numpy.cpp:199.) fgr_pose = torch.from_numpy(fgr_pose).to(self.device).unsqueeze(0).float() Saving metrics to /data/ldzhang/SSD/datasets/dreg_nerf/eval/train_nerf_regtr_obj/objaverse/4b05/transformation_est.json Traceback (most recent call last): File "/data/ldzhang/Projects/DReg-NeRF/eval_nerf_regtr.py", line 466, in <module> evaluator.evaluate() File "/data/ldzhang/Projects/DReg-NeRF/eval_nerf_regtr.py", line 238, in evaluate self.eval_iteration(data_batch=data_batch) File "/data/ldzhang/anaconda3/envs/dreg_nerf/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(*args, **kwargs) File "/data/ldzhang/Projects/DReg-NeRF/eval_nerf_regtr.py", line 349, in eval_iteration render_videos( File "/data/ldzhang/anaconda3/envs/dreg_nerf/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(*args, **kwargs) File "/data/ldzhang/Projects/DReg-NeRF/eval_nerf_regtr.py", line 136, in render_videos src_rgbs, src_depths = synthesize_novel_views( # In source nerf's coordinate frame. File "/data/ldzhang/anaconda3/envs/dreg_nerf/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(*args, **kwargs) File "/data/ldzhang/Projects/DReg-NeRF/eval_nerf_regtr.py", line 90, in synthesize_novel_views rgb, _, depth, _ = render_image( File "/data/ldzhang/Projects/DReg-NeRF/conerf/utils/utils.py", line 110, in render_image ray_indices, t_starts, t_ends = ray_marching( File "/data/ldzhang/anaconda3/envs/dreg_nerf/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(*args, **kwargs) File "/data/ldzhang/anaconda3/envs/dreg_nerf/lib/python3.9/site-packages/nerfacc/ray_marching.py", line 177, in ray_marching packed_info, ray_indices, t_starts, t_ends = _C.ray_marching( File "/data/ldzhang/anaconda3/envs/dreg_nerf/lib/python3.9/site-packages/nerfacc/cuda/__init__.py", line 13, in call_cuda return getattr(_C, name)(*args, **kwargs) RuntimeError: Trying to create tensor with negative dimension -1929448213: [-1929448213, 1]

Can you provide download links to the training data set and the trained nerf model

No module named 'conerf'

Thanks for your code. When I ran your code, I met the problem as follows,
from conerf.radiance_fields.ngp import NGPradianceField
ModuleNotFoundError: No module named 'conerf'

dataset devide

Can you provide the code for training two NERFs by clustering image data sets with KMeans？

transform files

Hi,

First of all, thank you for your work on this paper and dataset.
I tried to run NeRF on the evaluation dataset, but there are 5 scenes where the transform file makes no sense.
The rotation matrix values of the each frame is either too small or too high and I can't reproduce a NeRF out of it, the cameras location is not reasonable. Maybe there is a scale factor?

The scenes are:
Banana_3D_Scan2373
Banana_for_scale0a07
F_15_C_Jungle_Camo16c6
Procedural_Bananas3a07
Smoked_Salmon_Plate5648

Thanks