Comments (11)
It seems that you mix all the sequences from different scenes. You have to separate the sequences and reconstruct them independently using Colmap and NeRF. nerf_pl is a good starting point to learn NeRF.
from climate_nerf.
Hi @y-u-a-n-l-i,
Thanks for your reference.
I took the dataset from this source.
Here is what I did to generate the images and bin files:
- I specifically used images from the image_00 directory and provided them as input to the NerfStudio command to generate the images directory (images, images_2, images_4, images_8). I used the following command from NerfStudio for that:
ns-process-data images --data C:/Data_Science/Vikas_Projects/Custom_kitti/images/ --output-dir C:/Data_Science/Vikas_Projects/Custom_kitti/output/
-
Since the above Nerfstudio command didn't generate the bin files (it showed "no image pair found"), I edited the reconstruction options available in the COLMAP GUI, particularly for forward motion, which in turn generated the bin files inside the sparse directory. Here is a screenshot of the settings:
-
From the above process, I got the image directories from the first step and bin files required from the second step and used them as the dataset for training.
Thanks in advance for your help.
from climate_nerf.
You can refer to this processed data by Colmap. In the images
folder, there are only images from 1538 to 1601. Directly running our code on this data gives you rational results, but reconstructions will be better if you have lidar depth or use stereo images like the KITTI-360 dataset.
from climate_nerf.
I‘ve added a dataloader for Colmap processed dataset in MipNeRF360's format. Here is a breakdown of the directory structure:
├── images
├── images_2
├── images_4
├── images_8
├── poses_bounds.npy
└── sparse
poses_bounds.npy
is not necessary in our code. You need to run Colmap for your captures to get cameras.bin
, images.bin
and points3D.bin
in sparse/0
. Be careful when you are trying to load down-sampled images for training. For example, you need to store 4-times downsampled images in images_4
folder.
After that, you need to parse colmap
to --dataset_name
to enable the dataloader for Colmap during training and all the remaining steps. You can refer to this line in opt.py
for all the supported dataset types. You can also refer to the config file of Garden
scene in MipNeRF360
dataset for how our experiments are conducted on Colmap processed datasets.
from climate_nerf.
Hi, thanks for the reply. As you suggested, I tried running COLMAP on my custom video to generate the cameras.bin
, images.bin
, and points3D.bin
files in the sparse/0 directory. I managed to generate four folders named images, images_2, images_4, and images_8, along with a colmap directory containing a sparse directory which is empty. However, I am unable to generate the three bin files you mentioned. The process is showing an error that no good initial image pair is found.
My video features pure forward motion, and I discovered that pure forward motion is problematic for structure-from-motion as it leads to unstable (and degenerate) configurations. I am stuck at this point. If you have any idea about how to generate the bin files, it would be a great help to me.
Below is the link to my dataset.
Thanks in advance.
from climate_nerf.
Hi @y-u-a-n-l-i ,
I hope you are doing well. Following your suggestion to work on a custom dataset using Climate NeRF, I took images from the Kitti-360 data provided in the readme section and generated the dataset in the specified format using Colmap. After adjusting some reconstruction options, I successfully generated the bin files and sparse directory with Colmap. However, when I used this dataset to train the model with your training code, the reconstruction quality was not as expected compared to the original Kitti-360 dataset result. I will attach the render_traj.mp4 video output that I obtained when I trained the model with the dataset I built using Colmap.
Below is the link to my video.
Any assistance would be greatly appreciated.
from climate_nerf.
Sorry for the late response. It seems that I don't have the right to access your link.
It is recommended to use the dataloader designed for KITTI-360 if you want to run experiments on that dataset since we load KITTI-360's ground truth depths as geometry supervisions. You can find the example config file here and organize KITTI-360 data according to this.
python train.py --config configs/kitti360_1538.txt
is the command that I used to train our model.
from climate_nerf.
Hi @y-u-a-n-l-i ,
Thanks for your response. I'd like to clarify a few points:
-
I'm not working with the KITTI dataset. I have already trained a model using the KITTI-360 dataset following this approach.
-
What I did with the KITTI dataset was to use its images to create an images directory and bin files with COLMAP. However, whenever I tried to train the Climate NeRF model with the dataset I prepared using COLMAP, it didn't work. I will attach the video below.
render_traj.mp4
3.My main concern is running Climate NeRF on my custom dataset, which is very similar to the KITTI-360 dataset. I generated the images directory and bin files using Colmap as per your last reply, but whenever I try to train the model, I can't get a well-reconstructed video.
Any assistance would be greatly appreciated.
from climate_nerf.
It should be related to the scene scale parameter. Or Colmap failed to reconstruct camera poses on your sequence. Can you share your config file and your running command? Sending your processed data to me also helps. Also, if you are using Colmap gui, you might visualize and check the camera poses and sfm point cloud before running nerf code.
from climate_nerf.
Hi @y-u-a-n-l-i,
I am attaching a drive link where you can find the datasets (4 image directories and the sparse directory created using COLMAP), the configuration file, and the commands I used to train the model.
https://drive.google.com/drive/folders/17l5JT5KncQewyK-zukz-Dd2GJlpCcPoz?usp=sharing
Thanks in advance for your help.
from climate_nerf.
But have you ever really checked the images inside the dataset? You are trying to reconstruct images from different scenes in just one Colmap and NeRF model. This might lead to the failure of your optimization. There is no problem with your Colmap and image operations.
from climate_nerf.
Related Issues (17)
- About the dataset and training code for scenes in Rising Sea-level Simulation section. HOT 4
- How to “Download checkpoint of MTMT from official repo.”? HOT 1
- RuntimeError: expected scalar type Float but found Half HOT 2
- hardware requirements HOT 1
- Can I run this project on my PC with a Windows10 ? HOT 2
- the nan values during training ipdb HOT 2
- how to find 79_slim.ckpt in the playground.sh file? HOT 2
- ModuleNotFoundError: No module named 'vren' HOT 2
- RuntimeError: expected scalar type Float but found Half HOT 2
- plane.npy HOT 2
- The error in requirements.txt HOT 2
- Assertion `srcIndex < srcSelectDimSize` failed HOT 2
- RuntimeError: The size of tensor a (600000) must match the size of tensor b (552384) at non-singleton dimension 0
- How to compute "plane.npy"? HOT 1
- Problem:The version of mmengine,mmseg,mmcv HOT 1
- About the coordinate system using during rendering. HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from climate_nerf.