Visualizing bodyArray in RViz about k2_client HOT 5 CLOSED

@stepelle do you have any advice on how you normally use the package for skeleton tracking?

We normally do our visualization through RViz as well, but I'm not sure why your camera frame is is so far off. Do the extrinsics of the camera match?

from k2_client.

There is a good chance that I might be doing something wrong since this is my first time dealing with camera calibration and coordinate frames.

When you say the extrinsics match, I assume you mean the position and pose of the camera with respect to the fixed world frame? I have the camera at the origin. I figured that would be OK since I simply want an RViz overlay of the skeleton data over any of the image streams. I understand that the bodyArray values are in the kinect's local coordinate frame.

Let me explain what I've done. Perhaps you can figure out what I'm missing :)

When I tried using the camera topics without any intrinsic calibration data, RViz gave me the error message CameraInfo/P resulted in an invalid position calculation (nans or infs). This doesn't really surprise me since all the camera_info messages looked like this:

header: 
  seq: 11691
  stamp: 
    secs: 1457501451
    nsecs: 942718029
  frame_id: /head/kinect2/depth
height: 0
width: 0
distortion_model: ''
D: []
K: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
R: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
P: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
binning_x: 0
binning_y: 0
roi: 
  x_offset: 0
  y_offset: 0
  height: 0
  width: 0
  do_rectify: False

So then I tried using iai_kinect's calibration tool to get intrinsic values for the Kinect. This is a sample of what I got:

%YAML:1.0
rotation: !!opencv-matrix
   rows: 3
   cols: 3
   dt: d
   data: [ 9.9996239851124447e-01, -8.0104495945738600e-03,
       -3.3217858046474574e-03, 8.0603270685314973e-03,
       9.9985070338985749e-01, 1.5284045876520026e-02,
       3.1988577941922196e-03, -1.5310245853677917e-02,
       9.9987767435857500e-01 ]
translation: !!opencv-matrix
   rows: 3
   cols: 1
   dt: d
   data: [ -5.2086446196814358e-02, -9.1331596425510566e-05,
       1.2157507134049209e-02 ]
essential: !!opencv-matrix
   rows: 3
   cols: 3
   dt: d
   data: [ -9.8285640627523386e-05, -1.2154293750250829e-02,
       -2.7713632101032302e-04, 1.2323667128069865e-02,
       -8.9484339501056818e-04, 5.2039690054255995e-02,
       -3.2850562996227770e-04, -5.2079401474112358e-02,
       -7.9639501721752470e-04 ]
fundamental: !!opencv-matrix
   rows: 3
   cols: 3
   dt: d
   data: [ -1.3717142153371363e-07, -1.6913597158177371e-05,
       3.2403721265009977e-03, 1.7162226658549971e-05,
       -1.2425486406305027e-06, 2.2368510305431070e-02,
       -8.7367784202739846e-03, -6.0679132296321742e-02, 1. ]

But k2_client complained and spat out this error message: [ERROR] [1456976620.814128636]: Exception parsing YAML camera calibration: yaml-cpp: error at line 0, column 0: bad conversion.

As a YAML sanity check, I then went to the ROS Wiki and downloaded a sample calibration YAML with the following values:

image_width: 512
image_height: 424
camera_name: depth
camera_matrix:
  rows: 3
  cols: 3
  data: [4827.94, 0, 1223.5, 0, 4835.62, 1024.5, 0, 0, 1]
distortion_model: plumb_bob
distortion_coefficients:
  rows: 1
  cols: 5
  data: [-0.41527, 0.31874, -0.00197, 0.00071, 0]
rectification_matrix:
  rows: 3
  cols: 3
  data: [1, 0, 0, 0, 1, 0, 0, 0, 1]
projection_matrix:
  rows: 3
  cols: 4
  data: [4827.94, 0, 1223.5, 0, 0, 4835.62, 1024.5, 0, 0, 0, 1, 0]

This fixed all the k2_client and RViz error messages but to no one's surprise, markers created from /bodyArray values didn't align with the image at all. I tried formatting the iai_kinect YAML to look like the sample YAML from the wiki, but the parser error message came back. Hence my frustration.

Hope this provides some more context. Thanks!

from k2_client.

I use a package that subscribes the topic bodyarray and publish the body joint as a group of tf to be visualized in rviz. The bodyarray values are in the kinect reference system. The group of tf can be moved in the space on the basis of the kinect position in the space. However, I have never tried to match the information coming from the camera with the skeleton tracking.

from k2_client.

Interesting. What about you guys, @psigen? Have you tried matching pixels from the video to the bodyArray markers?

Also, I had the chance to revisit this issue after I sent both of you that long message. The reason for the YAML parser error is that CameraInfoManager::loadCameraInfo doesn't recognize parameters such as translation, rotation, and projection that are generated by the iai_kinect calibration program. On looking at the iai_kinect codebase, I see that they do a bunch of processing of their own YAML files before they publish their camera_info topic.

If you do side-by-side visualization of the camera and bodyArray, do you have a piece of code that can handle calibration files like that of the iai_kinect? Or a better question, how do both of you generate calibration files for your kinect?

from k2_client.

Just realized that I had left this open. @stepelle , thanks for explaining what was going on offline. Going to close this now

from k2_client.