Giter VIP home page Giter VIP logo

Comments (3)

dunbar12138 avatar dunbar12138 commented on May 23, 2024

Hi, thanks for your attention to our work!

Unfortunately, I cannot give you a direct answer for lack of information. I would say using some tools to visualize the cameras will help a lot when you debug.

from dsnerf.

mertkiray avatar mertkiray commented on May 23, 2024

Hello @dunbar12138 ,
Thank you for your help.
I tried to visualize the rays with the package you suggested.


import pytransform3d.visualizer as p
import pytransform3d.transformations as pt

index_pose = i_train[0]
rays_o, rays_d = get_rays_by_coord_np(H, W, focal, poses[index_pose,:3,:4], depth_gts[index_pose]['coord'])
rays_o, rays_d = torch.Tensor(rays_o).to(device), torch.Tensor(rays_d).to(device)


pose = poses[index_pose, :, :]
pose_hom = np.eye(4)
pose_hom[:3, :4] = pose
pose = pose_hom

fig = pv.figure()

cam2world = pose
virtual_image_distance = 1
sensor_size = np.array([W, H])
 intrinsic = [
    [focal, 0, sensor_size[0] / 2], [0, focal, sensor_size[1] / 2], [0, 0, 1]
]
intrinsic_matrix = np.array(intrinsic)

fig.plot_transform(A2B=cam2world, s=0.2)
fig.plot_camera(
    cam2world=cam2world, M=intrinsic_matrix, sensor_size=sensor_size,
    virtual_image_distance=virtual_image_distance)

for i in range(rays_d.shape[0]):
    ray_o = rays_o[i].cpu()
    ray_d = rays_d[i].cpu()

    fig.plot_vector(start=ray_o,
        direction=ray_d,
        c=(1.0, 0.5, 0.0))

    P = np.zeros((2, 3))
    colors = np.empty((2, 3))
    P[0] = ray_o

    # ray_o + (depth_gts[index_pose]['depth'][0] * rays_d)
    P[1] = ray_o + (depth_gts[index_pose]['depth'][i] * -ray_d)
    colors[:, 0] = np.linspace(1, 0, len(colors))
    colors[:, 1] = np.linspace(0, 1, len(colors))
    fig.plot(P, colors)

pcd = np.load('colmap_pointcloud.npy')

pcd_colmap = open3d.geometry.PointCloud()
pcd_colmap.points = open3d.utility.Vector3dVector(pcd)
pcd_colmap.paint_uniform_color([1, 0, 0])

pcd_colmap.transform(pose)
 fig.add_geometry(pcd_colmap)

fig.view_init()
if "__file__" in globals():
    fig.show()
else:
    fig.save_image("__open3d_rendered_image.jpg")

But the resulting visualization is like this:
image

Shouldn't the rays intersect with the points as we go draw a line from camera in the ray direction with the depth value?

Also, the rays are pointing in the negative direction of the camera.
image

Is this correct?

When I try the above implementation on my own dataset with known poses and point cloud I get a similar output.

image

which I think is wrong.

Can you please tell me what am I doing wrong :)

from dsnerf.

IncludeMathH avatar IncludeMathH commented on May 23, 2024

I'm also interested in this problem, and I think it may have something to do with the t vector in the transform matrix. I would appreciate it if you could provide your own discovery or solution.

from dsnerf.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.