Where do you use mono_depth? about ml-neuman HOT 7 OPEN

Awesome. thanks!

from ml-neuman.

The depth values are used for regularizing the background nerf, and you can find it inside the dataloader:

from ml-neuman.

Hi, I backtraced your code, and it seems you're not using mono_depth, but only use depth_maps.

1. Read depth data with from this code (https://github.com/apple/ml-neuman/blob/0149d258b2afe6ef65c91557bba9f874675871e4/train.py#L37C26-L37C26).
2. read_data_to_ram can be found in here:

   ml-neuman/scenes/scene.py

   Line 24 in 0149d25

   def read_data_to_ram(self, data_list=['image']):

3. read_data_to_ram function calls read_image_to_ram and read_depth_to_ram function for each cap, where cap is an instance of this class (

   ml-neuman/data_io/neuman_helper.py

   Line 22 in 0149d25

   class NeuManCapture(captures_module.RigRGBDPinholeCapture):

   )
4. read_image_to_ram first reads mono_depth with this code (

   ml-neuman/data_io/neuman_helper.py

   Line 52 in 0149d25

   return self.captured_image.read_image_to_ram() + self.captured_mask.read_image_to_ram() + self.captured_mono_depth.read_depth_to_ram()

   )
5. But it overwrites the depth data with read_depth_to_ram again from the above 3rd stage, where the read_depth_to_ram calls this function (

   ml-neuman/cameras/contents.py

   Line 110 in 0149d25

   def read_depth(self):

   ).
6. As captured_depth is defined with depth_path, not mono_depth_path, it overwrites mono_depth data of the above 4th stage with depth data of the 5th stage. FYI, depth_path refers to MVS depth data.

Could you check am I right? Thanks!

from ml-neuman.

Sorry for the over complicated pipeline... It was inherited from a sfm project.
I think what happens to mvs depth and monocular depth is that we used monocular depth to fill the holes in mvs depth maps. see:

from ml-neuman.

Great thanks! Now I got it. BTW, are you using other geometry data, such as densepose and keypooints when training NeRF (not for preprocessing)? It seems NeuManCapture loads them but not use them

from ml-neuman.

We didn't use densepose/keypoints during nerf training stage, iirc. You can double check by setting them to None manually.

from ml-neuman.

Hi jianwei221, two follow-up questions.

1. Why do you use mono_depth only for background? Why not use it for foreground?
2. Why do you adjust scale and translation only based on human area like this (

   ml-neuman/data_io/neuman_helper.py

   Line 83 in 0149d25

   res = scipy.stats.linregress(x, y)

   )? Why not use bkg area for the adjustment?

Thanks in advance!

from ml-neuman.