mizhenxing / switch-nerf Goto Github PK
View Code? Open in Web Editor NEWCodes for Switch-NeRF (ICLR 2023)
Home Page: https://mizhenxing.github.io/switchnerf
License: MIT License
Codes for Switch-NeRF (ICLR 2023)
Home Page: https://mizhenxing.github.io/switchnerf
License: MIT License
Hello author, thank you for opening source the code for this work. When I tested the model, I found that it went through a round of iterations for about 113s, with a total of 20 rounds. May I ask if this 113s is the time used to render a picture? My device is RTX 3090
Thanks for your excellent work!
I noticed the visualized results are all experiments of the drone view scenes.
Is there any visualized results on scenes like block-nerf data?
As the title.
Hello!
Thank you for sharing your code.
I am trying to reproduce your result on Mega-NeRF building dataset.
I am using one RTX A6000 GPU.
During generating chunks, I have an error.
Traceback (most recent call last):
File "/home/user/anaconda3/envs/nerf/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/user/anaconda3/envs/nerf/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/user/switchnerf/Switch-NeRF/switch_nerf/train.py", line 28, in <module>
main(_get_train_opts())
File "/home/user/anaconda3/envs/nerf/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper
return f(*args, **kwargs)
File "/home/user/switchnerf/Switch-NeRF/switch_nerf/train.py", line 24, in main
Runner(hparams).train()
File "/home/user/switchnerf/Switch-NeRF/switch_nerf/runner.py", line 532, in train
dataset = FilesystemDataset(self.train_items, self.near, self.far, self.ray_altitude_range,
File "/home/user/switchnerf/Switch-NeRF/switch_nerf/datasets/filesystem_dataset.py", line 55, in __init__
append_arrays = self._check_existing_paths(chunk_paths, center_pixels, scale_factor,
File "/home/user/switchnerf/Switch-NeRF/switch_nerf/datasets/filesystem_dataset.py", line 296, in _check_existing_paths
assert (chunk_path / 'metadata.pt').exists(), \
AssertionError: Could not find metadata file (did previous writing to this directory not complete successfully?)
where can I generate metadata.pt?
I thought that the example script generates the data chunk, but somehow it requires me to give it metadata of the data chunk.
I am trying to run Mega-NeRF too, but it is hard to find.
Please let me know if you have any solution.
Also, in the bungee-nerf training code,
CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --use_env --master_port=12345 --nproc_per_node=1 -m mega_nerf.train_nerf_moe --config=switch_nerf/configs/switch_nerf/bungee.yaml
So there comes "mega_nerf.train_nerf_moe FAILED" error.
I guess it should be changed to "switch_nerf.train_nerf_moe" ! and it works well.
Thank you!
I use 2 RTX3090 for block-nerf dataset training, and I find that it runs out of memory in moe module. Is that normal?
Althought I decrease the batch_size
, even to batch_size=2312
, it still went out of memory.
And my scripts are like:
CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch \
--use_env --master_port=12345 --nproc_per_node=2 -m \
switch_nerf.train \
--config=switch_nerf/configs/switch_nerf/mission_bay.yaml \
--use_moe --exp_name=data/outputs \
--dataset_path=data/datasets/waymo-block-nerf/v1.0 \
--block_train_list_path=switch_nerf/datasets/lists/block_nerf_train_val.txt \
--block_image_hash_id_map_path=switch_nerf/datasets/lists/block_nerf_id_map.json \
--chunk_paths=data/chunks/mission_bay_chunk_radii_1 \
--no_bg_nerf --near=0.01 --far=10.0 \
--use_balance_loss \
--i_print=200 \
--batch_size=12312 \
--moe_expert_type=expertmlp \
--moe_train_batch \
--moe_test_batch \
--model_chunk_size=212992 \
--coarse_samples=257 \
--fine_samples=257 \
--moe_capacity_factor=1.0 \
--batch_prioritized_routing \
--moe_l_aux_wt=0.0005 \
--amp_use_bfloat16 \
--use_moe_external_gate \
--use_gate_input_norm \
--use_sigma_noise \
--sigma_noise_std=1.0
Hi, I followed readme and tried this example code, but got an error: 'Error while finding module specification for 'mega_nerf.train_nerf_moe' (ModuleNotFoundError: No module named 'mega_nerf')'. And I didn't find the mega_nerf directory nor train_nerf_moe.py file. Are there any files missed?
Hi.
Thank you for your work.
What should I prepare for training my custom dataset?
I have converted my dataset to the mega_nerf format and used meta_nerf style commands(building.yaml) to generate chunks, but the results is so bad and i think i miss some important steps when i prepare custom dataset. Do you have any scripts or tips for preparing custom dataset?
I downloaded the provided ckpt of building,then ran the following command:
CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --use_env --master_port=12345 --nproc_per_node=1 -m switch_nerf.eval_image --config=switch_nerf/configs/switch_nerf/building.yaml --use_moe --exp_name=output/building-pixsfm/ --dataset_path=data/building-pixsfm/ --i_print=1000 --moe_expert_type=seqexperts --model_chunk_size=131072 --ckpt_path=output/building-pixsfm/ckpt/building.pt --expertmlp2seqexperts --use_moe_external_gate --use_gate_input_norm
The evaluation indicators obtained are as follows:
INFO:root:Average val/time: 44.57523021697998 2023-11-29 19:59:28,001 root INFO: Average val/time: 44.57523021697998 INFO:root:Average val/memory: 5856.643969726562 2023-11-29 19:59:28,001 root INFO: Average val/memory: 5856.643969726562 INFO:root:Average val/psnr: 21.543164134025574 2023-11-29 19:59:28,001 root INFO: Average val/psnr: 21.543164134025574 INFO:root:Average val/ssim: 0.5792561799287796 2023-11-29 19:59:28,001 root INFO: Average val/ssim: 0.5792561799287796 INFO:root:Average val/lpips/vgg: 0.47358196824789045 2023-11-29 19:59:28,001 root INFO: Average val/lpips/vgg: 0.47358196824789045 INFO:root:Average val/lpips/alex: 0.39700327515602113 2023-11-29 19:59:28,001 root INFO: Average val/lpips/alex: 0.39700327515602113 INFO:root:Average val/lpips/squeeze: 0.29448841512203217 2023-11-29 19:59:28,001 root INFO: Average val/lpips/squeeze: 0.29448841512203217
These results are consistent with the results presented in the paper on the building data.However, when I looked at the rendered image, I felt that it was not very good and there were some artifacts and blurring.
Is this caused by parameter settings? Thank you for your help.
What if I want to visualize a specified expert output, to see how the input scene be decomposed by Switch-NeRF's gating network?
I was working on the research of NeRF. I've noticed your submission on 2023 ICLR and it is a great honour for me to read your article, Switch-NeRF: Learning Scene Decomposition with Mixture of Experts for Large-scale Neural Radiance Fields. It is impressive that you bring MoE into the application of NeRF. The work makes me refreshed and I want to induct it into other fields. I am very interested in your research and want to further explore the research. But unfortunately, I can not access the codes on Github URL you leave when I try to implement your algorithm. Could you please upload the source code of your project?
Hi, @MiZhenxing
May I know how you generate the camera poses to render the demo?
[E ProcessGroupNCCL.cpp:587] [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1801171 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:587] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1801172 milliseconds before timing out.
Why is this?
This seems to indicate that a timeout error occurred while performing some kind of parallel computing task. Specifically, when using NCCL (NVIDIA Collective Communications Library) for communication, a certain operation took longer than the set threshold (1800000 milliseconds, or 30 minutes) and was therefore interrupted.
Have you ever encountered such a problem? Can you give me some solutions?
Hi,
Thank you for the exceptional code you provided! However, after running 500,000 iterations on the Building dataset and attempting to visualize the merged ".ply" file, I encountered unsatisfactory results in Meshlab, as illustrated below.
Can you assist me in identifying the reason behind the occurrence of this problem, please? Thanks!
I cannot build tutel correctly, did you ever met the problem that cannot find 'tutel_custom_kernel'?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.