Comments (7)
Awesome. thanks!
from ml-neuman.
The depth values are used for regularizing the background nerf, and you can find it inside the dataloader:
ml-neuman/datasets/background_rays.py
Line 51 in 0149d25
from ml-neuman.
Hi, I backtraced your code, and it seems you're not using mono_depth
, but only use depth_maps
.
- Read depth data with from this code (https://github.com/apple/ml-neuman/blob/0149d258b2afe6ef65c91557bba9f874675871e4/train.py#L37C26-L37C26).
read_data_to_ram
can be found in here:Line 24 in 0149d25
read_data_to_ram
function callsread_image_to_ram
andread_depth_to_ram
function for each cap, where cap is an instance of this class (ml-neuman/data_io/neuman_helper.py
Line 22 in 0149d25
read_image_to_ram
first readsmono_depth
with this code (ml-neuman/data_io/neuman_helper.py
Line 52 in 0149d25
- But it overwrites the depth data with
read_depth_to_ram
again from the above 3rd stage, where theread_depth_to_ram
calls this function (Line 110 in 0149d25
- As
captured_depth
is defined withdepth_path
, notmono_depth_path
, it overwrites mono_depth data of the above 4th stage with depth data of the 5th stage. FYI,depth_path
refers to MVS depth data.
Could you check am I right? Thanks!
from ml-neuman.
Sorry for the over complicated pipeline... It was inherited from a sfm project.
I think what happens to mvs depth and monocular depth is that we used monocular depth to fill the holes in mvs depth maps. see:
ml-neuman/data_io/neuman_helper.py
Lines 77 to 86 in 0149d25
Then the fused depth map is used to regularize the bkg nerf:
ml-neuman/datasets/background_rays.py
Line 82 in 0149d25
from ml-neuman.
Great thanks! Now I got it. BTW, are you using other geometry data, such as densepose and keypooints when training NeRF (not for preprocessing)? It seems NeuManCapture
loads them but not use them
from ml-neuman.
We didn't use densepose/keypoints during nerf training stage, iirc. You can double check by setting them to None manually.
from ml-neuman.
Hi jianwei221, two follow-up questions.
- Why do you use mono_depth only for background? Why not use it for foreground?
- Why do you adjust scale and translation only based on human area like this (
ml-neuman/data_io/neuman_helper.py
Line 83 in 0149d25
Thanks in advance!
from ml-neuman.
Related Issues (20)
- Problem with preprocessing my own video HOT 3
- "Error: Cannot Handle Data Type: (1, 1, 3), <f4" HOT 1
- The loss is really small,but the result is weird HOT 8
- 你的yml文件安装报错,输入你的分开安装的代码,也是报错,怎么回事?
- Question about the empty_space_loss
- Questions about data preprocessing HOT 1
- the new data set training problem
- Question about Telegathering
- Training human nerf model problem
- 为什么我运行demo生成的图像都是黑色的? HOT 1
- Preprocessing Dockerfile Doesn't Build HOT 3
- Training too slow
- Encounter version conflict when building the docker environment. HOT 1
- Colab
- res101.pth can't download HOT 1
- colmap exit
- The quality of the scene in reposing images are bad
- About the issue of rendering new poses HOT 1
- Output
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ml-neuman.