IMU + RGBD image drift about rtabmap HOT 5 OPEN

Can you share a rosbag of the following topics?

rosbag record /tf /tf_static \
   /input/sensor/rpi_camera_v3_wide_384_216/image_rect_color \
   /input/sensor/rpi_camera_v3_wide_384_216/camera_info \
   /input/sensor/rpi_camera_v3_wide_384_216/depth_image_0_gen_rect \
   /input/sensor/imu/data_processed \
   /royale_cam_left_lidar/point_cloud_0

Here are some stuff that can go wrong:

• Low or different frame rate between the RGB and depth cameras
• Bad synchronization of the rgb and depth frames
• Bad TF between the RGB and depth cameras (how extrinsics have been computed?)
• ...

from rtabmap.

Yes, here you go: https://drive.google.com/file/d/1bxXn9blpWZCKAzm6dPpivV9nQd-p6DIQ/view?usp=sharing

I have only done manual calibration of the RGB and depth camera. Do you have a good library to do the extrinsic calibration?

from rtabmap.

Can you have an IR stream from the depth camera? If so, you may be do something like this: http://wiki.ros.org/openni_launch/Tutorials/ExtrinsicCalibrationExternal

It seems there is also a module in opencv to do that: https://docs.opencv.org/4.x/d2/d1c/tutorial_multi_camera_main.html

For the data in the rosbag, thetime difference between RGB and depth frames is sometime too large:

The time difference between rgb and depth frames is high (diff=0.133255s, rgb=1709536481.863311s, depth=1709536481.730056s).

It is a problem when the camera is moving, the depth image won't be correctly registered to color image. On Google Tango we had the same issue (depth image frame rate is 5 Hz and rgb camera is 30 Hz), but with Tango VIO computed at 30 Hz, we could correctly register the depth cloud to corresponding rgb frame.

The depth is not dense enough for feature extraction, either reduce the depth resolution by two (to make it more dense) or fill the holes:

For such small resolution, you can set GFTT/MinDistance to 3.

For reference, I tested the bag with:

roslaunch rtabmap_launch rtabmap.launch \
   args:="-d --GFTT/MinDistance 3 --Rtabmap/DetectionRate 0" \
   rgb_topic:=/input/sensor/rpi_camera_v3_wide_384_216/image_rect_color \
   camera_info_topic:=/input/sensor/rpi_camera_v3_wide_384_216/camera_info \
   depth_topic:=/input/sensor/rpi_camera_v3_wide_384_216/depth_image_0_gen_rect  \
   use_sim_time:=true \
   frame_id:=elp_camera \
   queue_size:=50

from rtabmap.

Thank you so much for such good support. I am already getting way better results by increasing the density of the density image.
One thing I am wondering about is why my odom frame is rotating compared to the map frame. It starts off really good where both odom and map frame are almost the same:

But it then suddenly rotates and shifts:

Shouldn't the IMU gravity constraint how the odom frame is allowed to move with regards to the map frame? No mather how bad the relation between odom and map is it seems like the relation between base_link and odom is correct(orientation and position).

from rtabmap.

Do you delete the database on each restart (argument --delete_db_on_start)? It looks like it is relocalizing on a previous session. Otherwise, map -> odom would change on loop closure detection. In rtabmap_viz you would see a green background on loop closure view when it happens. Another thing that could happen is if the base_link->IMU transform is wrong, the map optimizer may wrongly optimize the map with IMU constraints. You can turn off IMU optimization to compare if it is the case by setting --Optimizer/GravitySigma 0.

from rtabmap.