There is a problem in image warping on LLFF dataset. about sinnerf HOT 12 CLOSED

Hi @skyir0n, It's weird. Can you please share the warping error you observe?

from sinnerf.

Thank you for the prompt reply. Unfortunately, I couldn't share it with images due to the company policy. The warping error that I've observed maybe due to geometry information such as inconsistent coordinates or inaccurate depth maps. Image misalignment is dominant between the warped and target (GT) images.

When it is observed that the problem occurred only in the LLFF dataset, I can guess the main reason depends on the geometry. In your code, the matrix, namely ww, is applied to the inversion of a camera pose matrix and is different according to datasets. Is it possible that this problem is related to the matrix multiplication for the coordinate transformation? Please check it out.

from sinnerf.

Sorry I didn't get your point. Are you trying the pipeline on other data from LLFF dataset?

from sinnerf.

I have been trying to apply your warping code to the LLFF dataset. By using 'fern' image and the depth map that you provided, I warped an image to the target image, both of which are in the testset. I also used the function named ‘forward_warping’ by importing “SinNeRF/datasets/llff_ray_patch_1image_proj.py”. However, the warped and the target images (GT) are not matched and misaligned when I compared the two images.

from sinnerf.

ok, i see. I haven't experimented on fern, actually. It might be because the depth map is wrong. Currently, the depth map is obtained via a pre-trained NeRF. Another issue might be occlusion, current painter algorithm has some issues. I'll try to run on fern maybe next week and see what the result looks like. Thanks for letting me know

from sinnerf.

I'd like to point out that this problem is not related to occlusion or painter algorithm. If partial warping error is observed, you are right. However, since the whole geometry is not aligned when comparing the warped and target images, there would be problem in depth map or coordinates. In addition, not only 'fern' but also other data in the LLFF dataset shows the same problem.

from sinnerf.

is this happening for room or flowers?

from sinnerf.

Yes. This problem is observed at all the data in the LLFF dataset.

from sinnerf.

are you using ndc for LLFF? my current implementation doesn't use ndc. I couldn't reproduce what you described. If you provide more visual results, I might be able to help more. Thank you!

from sinnerf.

from sinnerf.

Hi, I met the same issue on llff dataset, all warping results seems shift compared with target image , I wonder how do you solve this problem ?

from sinnerf.

@violet257 maybe you can check if ndc is enabled, also can validate the pose / warping code with some ground truth depth first

from sinnerf.