Discussion about bad results on scene horns of LLFF dataset about dsnerf HOT 2 OPEN

Hi, thank you for the reply!

I think it is still unclear why using depth supervision may give us suboptimal results, especially in the case of using more than 20 images. I have also experimented on scene fortress, which has a more compact, bounded scene space (less free space compared with scene horns). I use all images (as before) in this experiment. And the results are:

As expected, the results of 'no ndc only' is still worse than 'original'. On the other hand, unlike scene horns, depth supervision helps the PSNR number (compared with 'no ndc only'). So I am wondering whether depth supervision mainly works on scenes like fortress, which has a more compact, uniform, bounded space for sampling. Also, maybe selecting a better sampling strategy for scenes like horns, which has a larger depth range (unbounded) and much more free space, will prove the effectiveness of depth supervision on these scenes?

By the way, I find that no_ndc is alwasy set to true in your recently uploaded config files. Have you tried depth supervision with no_ndc=False, i.e., use depth supervsion in ndc space? I am not familiar with ndc space, but I think it kinds of re-parameterize the scene space to a more compact and uniform, and bounded spcae for sampling, and this maybe helpful for scenes like horns?

Hope for your reply!

from dsnerf.

Hi, thanks for your interest!

We also sometimes observe suboptimal PSNR when training DS-NeRF on more than 20 images. We suspect this is because the target functions of reconstructing depths and RGB don't always coincide especially when we're asking it to reconstruct fine-grained texture. Adding depth supervision when you already have sufficient training views might not give you as good RGB reconstruction, but would produce more reasonable depths and mesh. We're still looking for a good way to make depth supervision work better in the full-view setting.

Happy to discuss more if you are still interested!

from dsnerf.