Hello all, I am getting this warning that I think is affecting the m

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

FYI we should fix this with <a class="issue-link js-issue-link" data-error-text="Faile

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

Also if you are using our data.bag this rosbag does not contain a <code class="notrans

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

Getting the "Using latest TF transform instead of timestamp match." warning and not sure why...,about ethz-asl/voxblox

Comments (12)

ZacharyTaylor commented on May 31, 2024

There are two possible problems here.

To start with it could be an issue with the time sync as you are guessing. The warning you are seeing occurs when the timestamp of the current pointcloud has occurred after the timestamp of the last TF transform received. Because the system won't extrapolate into the future it just uses the last TF it has instead. This is usually an ok approximation for high rate sensors.

However, I feel the more likely issue will be that the transformation from the frame given by your vicon markers to your sensor frame is not quite setup correctly. What process have you been using to calibrate your extrinsics?

from voxblox.

helenol commented on May 31, 2024

Also as a small aside to what Zac said, you don't need to artificially slow down the FPS of your sensor -- simply set the min_time_between_msgs_sec parameter to drop all messages coming faster than that: <param name="min_time_between_msgs_sec" value="0.20" />

…

On Sat, Aug 11, 2018 at 12:08 AM Zachary Taylor ***@***.***> wrote: There are two possible problems here. To start with it could be an issue with the time sync as you are guessing. The warning you are seeing occurs when the timestamp of the current pointcloud has occurred after the timestamp of the last TF transform received. Because the system won't extrapolate into the future it just uses the last TF it has instead. This is usually an ok approximation for high rate sensors. However, I feel the more likely issue will be that the transformation from the frame given by your vicon markers to your sensor frame is not quite setup correctly. What process have you been using to calibrate your extrinsics? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#188 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFWzCELvutLoCOxrxfRHNeq7Wlee2dIoks5uPgRQgaJpZM4V4cgf> .

from voxblox.

Metalzero2 commented on May 31, 2024

@ZacharyTaylor
Related to your first point:
I see what you mean however the timestamp of the current pointcloud is never further then the latest TF timestamp. I wrote a small program that has two subscribers (one at the poinCloud publisher and the other to the TF publisher) and print the header.stamp information it has. Here is a small part of the output:

[ INFO] [1534242887.197223388]: TransformStaped Header:
[ INFO] [1534242887.197237548]: time stamp: sec->[1473521340], nsec->[711812753]
[ INFO] [1534242887.197253826]: TransformStaped Header:
[ INFO] [1534242887.197266720]: time stamp: sec->[1473521340], nsec->[721967786]
[ INFO] [1534242887.522669063]: PointCloud Header:
[ INFO] [1534242887.522742652]: time stamp: sec->[1473521340], nsec->[567782544]
[ INFO] [1534242887.527584895]: PointCloud Header:
[ INFO] [1534242887.527646920]: time stamp: sec->[1473521340], nsec->[635118454]
[ INFO] [1534242887.531881337]: PointCloud Header:
[ INFO] [1534242887.531942925]: time stamp: sec->[1473521340], nsec->[667177719]
[ INFO] [1534242887.531982634]: TransformStaped Header:
[ INFO] [1534242887.531996635]: time stamp: sec->[1473521340], nsec->[732776665]
[ INFO] [1534242887.532017508]: TransformStaped Header:
[ INFO] [1534242887.532031813]: time stamp: sec->[1473521340], nsec->[741815087]
[ INFO] [1534242887.532052045]: TransformStaped Header:
[ INFO] [1534242887.532063461]: time stamp: sec->[1473521340], nsec->[752854504]
[ INFO] [1534242887.532077295]: TransformStaped Header:
[ INFO] [1534242887.532087941]: time stamp: sec->[1473521340], nsec->[762102616]
[ INFO] [1534242887.532103516]: TransformStaped Header:
[ INFO] [1534242887.532114376]: time stamp: sec->[1473521340], nsec->[773670894]
[ INFO] [1534242888.093924904]: TransformStaped Header:
[ INFO] [1534242888.093973173]: time stamp: sec->[1473521340], nsec->[782622246]
[ INFO] [1534242888.093993936]: TransformStaped Header:
[ INFO] [1534242888.094009024]: time stamp: sec->[1473521340], nsec->[791702255]
[ INFO] [1534242888.094029067]: TransformStaped Header:
[ INFO] [1534242888.094043447]: time stamp: sec->[1473521340], nsec->[802608027]
[ INFO] [1534242888.094059937]: TransformStaped Header:
[ INFO] [1534242888.094073116]: time stamp: sec->[1473521340], nsec->[812573193]
[ INFO] [1534242888.304594486]: PointCloud Header:
[ INFO] [1534242888.304642267]: time stamp: sec->[1473521340], nsec->[700119594]
[ INFO] [1534242888.304675640]: TransformStaped Header:
[ INFO] [1534242888.304691447]: time stamp: sec->[1473521340], nsec->[822593778]
[ INFO] [1534242888.304707420]: TransformStaped Header:
[ INFO] [1534242888.304722317]: time stamp: sec->[1473521340], nsec->[832856896]

as you can see, the PoinCloud has a timestamp earlier then the latest TF timestamp. The only thing that happens is that the timestamp of the PointCloud is never exactly the same with a timestamp in the TF. There is always some small difference in milliseconds but that should be expected, correct?

Related to the second point:
I don't completely follow what you mean. For the calibration of my RealSense camera, I followed the instruction from Intel's guide (found here). However, I am not using my own camera but the data from the .dag file. The way I understand it, extrinsic settings are related with the physical camera the time it is capturing data. Are there extrinsic settings related to the pointCloud data in the .dag file?

@helenol
Indeed. Thanks for the tip. Will do!

from voxblox.

ZacharyTaylor commented on May 31, 2024

I looked it up and tf by default stores the last 10 seconds worth of transforms (http://docs.ros.org/jade/api/tf/html/c++/classtf_1_1TimeCache.html#acca87a70aeb9c3573cdbfec26f6bfe23) so the slight time offset between the two data streams isn't the issue. Are you seeing this warning on every frame or just some of them?

For the calibration I am a little unclear on a few things. In your first post you mentioned an IMU but your rosgraph shows you using vrpn and a depth sensor. What is your actual sensor setup, are you using a motion capture system for your transforms? Also what is a .dag file? Finally are you using data you have collected yourself or is this a publicly available dataset?

from voxblox.

helenol commented on May 31, 2024

FYI we should fix this with #201
it will now queue your incoming pointclouds until the transform is available.

from voxblox.

Metalzero2 commented on May 31, 2024

@ZacharyTaylor
Apologies for the delay. I was "off the grid" for a week.

Yes, I was aware of the 10 second buffer that tf has my default. The thing I was not sure of is this: after storing transformations in the tf, do I need the exact time stamp as to get a transformation or does tf just give you the closes entree relative to the requested time stamp (assuming you are with in the 10 sec buffer)? For example, let us say I have entrees at nsec= [912339188], [923085185] and [932359819]. If I request a transformation with nsec = [923090000] will it not return anything or will it return the closes transformation (which is the one with nsec = [923085185] in this example). From my understanding, it returns the closest one, which would make sense since having the exact same time stamp, in nsec detail, is close to impossible I think (especially in ROS).

To answer your first question, the warning is appearing in every frame.

Now for the second point you make. The .dag file was in reference to the "data.bag" file that you provided here. So I am not using my own data yet. From now own, to avoid confusing, when referring to data.dag file, I am referring to your provided data set. My setup is basically using the provided Voxblox with use_tf_transforms set to true. In other words, using the tf for the transformation. In the graph I provided, the "/camera/depth_registered/points" and "/kinect/vrpn_client/estimated_transformation" are the topics created with running your "player" node.

The idea is that after I make the Voxblox working with the tf, using your data set, I can replace the RGB-D and the transformation data from the data.dag, with my own RGB-D sensor and transformation data.

I hope this clears up what I am trying to do and what the problem is.

from voxblox.

helenol commented on May 31, 2024

@Metalzero2
TF does interpolation (I guess linear?) internally. So if you request 923090000, which is 912339188 < x < 932359819, it will interpolate between the nearest poses that are available.

What we did before (current master) was if your queue had times 0, 1, 2, 3 in there, and your new data coming in was at t=3.25, we would just use the pose at 3 as is (either from tf or transforms).
What we've done in PR #201 is queue your pointclouds until time 4 is available, and then interpolate (either through tf or our own transform queue) to get the best estimate at 3.25, and integrate the pointcloud then.

Edit: quick reference on TF interpolation: https://answers.ros.org/question/10285/how-does-tf-do-interpolation/

from voxblox.

ZacharyTaylor commented on May 31, 2024

Also if you are using our data.bag this rosbag does not contain a /tf topic. This means there is no pose for the system to look up through tf and so the bag can only be used with stamped transform messages. This means you must set use_tf_transforms = false.

Alternatively if you absolutely needed to use the tf tree with this bag, you could also run the following node to do the conversion https://github.com/ethz-asl/transform_tools

from voxblox.

Metalzero2 commented on May 31, 2024

@helenol
I was not aware if TF does interpolation or not because it was not written anywhere in its documentation (as far as I know). So thanks for mentioning it.

I was able to find the problem. I was creating the TF map with incorrect time stamps. So when a point cloud would come, it would be out of range of the TF map, which resulted in me getting that warning.

However, even though now I don't get that warning, the results of the Voxblox, using the data.dag file you provide, still does not resemble the results when having use_tf_transforms = false. The reason I insist on trying to use your data-set (data.dag) with the TF @ZacharyTaylor , is because I want to make sure that the Voxblox is working properly when using the TF map. Then, I can easily integrate it in my own project.

@ZacharyTaylor I have already implemented a node that does the conversion, which can be found here. It should work properly I believe.

Is it possible that the interpolation method used can have an effect on the results? Also, have you ever been able to run the Voxblox, with your data-set, while using TF and get the same results as when not using TF?

from voxblox.

ZacharyTaylor commented on May 31, 2024

I looked into the issue and it is again an issue with the tfs that is a little unique to the setup used for the cow and lady dataset. The transform given is the pose of the Vicon markers on the sensor, not the sensor itself. Because of this you need to apply a second transform to go from the marker frame to the sensor frame. When using transforms this is done via setting the parameter T_B_D (set in the cow_and_lady.yaml file). This is then applied internally in Voxblox.

When using a tf tree it is assumed that this static transform is part of the tf tree and so no additional internal transform is done. Because of this you will need to manually create a static tf publisher with the correct transformation parameters.

For example the following modified cow_and_lady launchfile creates and uses the needed tfs

<launch>
  <arg name="play_bag" default="true" />
  <arg name="bag_file" default="/home/z/Downloads/data.bag"/>
  <arg name="voxel_size" default="0.05"/>

  <node name="player" pkg="rosbag" type="play" output="screen" args="-r 1.0 --clock $(arg bag_file)" if="$(arg play_bag)"/>

  <param name ="/use_sim_time" value="true"/>

  <node name="transform_to_tf_node" pkg="transform_to_tf" type="transform_to_tf_node" output="screen">
    <remap from="transform_to_be_converted" to="/kinect/vrpn_client/estimated_transform"/>
    <param name="global_frame_name" value="world"/>
    <param name="local_frame_name" value="markers"/>
  </node>

  <node pkg="tf" type="static_transform_publisher" name="markers_sensor_broadcaster" args="0.00114049 0.0450936 0.0430765 0.0924132, 0.0976455, 0.0702949, 0.9884249 markers camera_rgb_optical_frame 100" />

   <node name="voxblox_node" pkg="voxblox_ros" type="tsdf_server" output="screen" args="-alsologtostderr" clear_params="true">
    <remap from="pointcloud" to="/camera/depth_registered/points"/>
    <param name="tsdf_voxel_size" value="$(arg voxel_size)" />
    <param name="tsdf_voxels_per_side" value="16" />
    <param name="voxel_carving_enabled" value="true" />
    <param name="color_mode" value="color" />
    <param name="use_tf_transforms" value="true" />
    <param name="update_mesh_every_n_sec" value="1.0" />
    <param name="min_time_between_msgs_sec" value="0.0" />
    <param name="method" value="fast" />
    <param name="use_const_weight" value="false" />
    <param name="allow_clear" value="true" />
    <param name="verbose" value="true" />
    <remap from="transform" to="/kinect/vrpn_client/estimated_transform" />
    <rosparam file="$(find voxblox_ros)/cfg/cow_and_lady.yaml"/>
    <param name="mesh_filename" value="$(find voxblox_ros)/mesh_results/$(anon cow).ply" />
  </node>

</launch>

from voxblox.

Metalzero2 commented on May 31, 2024

@ZacharyTaylor yes that was it. This static transformation is something I did not notice. I integrated that transformation inside my node that updates the tf map and it works properly.

Thank you a lot for your help!

from voxblox.

nbansal90 commented on May 31, 2024

@ZacharyTaylor I am encountering a an issue. Where everything I am able to generate semantic pointcloud, But 3D geometric reconstruction is not happening successfully and thus I am not able to visualized it rviz. I think I am on the same boat, in terms that, I have ground truth pose (4 x 4) matrix available with me, (which would provide us with relative movement between consecutive frames), and I had saved it in my ROSBAG file.(along with Rob, depth and labels)

Now as suggested in previous post, I have set the parameter use_tf_transforms to false before the reconstruction step in the launch file and subscribed to transform. I changed the existing kimera_semantics.launch file. to add my topics namely rgb_image_raw , depth_image_raw , labels_image_raw and orb_slam_pose

<?xml version="1.0" encoding="ISO-8859-15"?>
<launch>
  <arg name="voxel_size"     default="0.05"/>
  <arg name="max_ray_length_m" default="5"/>
  <arg name="should_use_sim_time" default="true" />
  <param name="use_sim_time" value="$(arg should_use_sim_time)" />

  <!-- Change sensor frame to:
   - 1. VIO's estimated base_link: `left_cam_base_link`
   - 2. Or, if you want to use simulator's ground-truth: `left_cam`
  -->
  <arg name="sensor_frame" default="left_cam"/>

  <!-- If you want to play directly from a rosbag -->
  <arg name="play_bag" default="false"/>
  <!-- Let's go at twice real-time to show the speed! It can go faster, but then rviz struggles. -->
  <arg name="rosbag_rate" default="1.0"/>
  <arg name="bag_file" default="$(find kimera_semantics_ros)/rosbags/scannet_rosbag.bag"/>

  <!-- If you just want to run 3D reconstruction without semantics, set this flag to false-->
  <arg name="metric_semantic_reconstruction" default="true"/>
  <arg name="semantic_label_2_color_csv_filepath" default="$(find kimera_semantics_ros)/cfg/tesse_multiscene_office1_segmentation_mapping.csv"/>

  <!-- Input -->
  <arg name="semantic_pointcloud"         default="/semantic_pointcloud"/>
  <arg name="left_cam_info_topic"         default="/tesse/left_cam/camera_info"/>
  <arg name="right_cam_info_topic"        default="/tesse/right_cam/camera_info"/>
  <arg name="left_cam_topic"              default="/tesse/left_cam/image_raw"/>
  <arg name="right_cam_topic"             default="/tesse/right_cam/image_raw"/>
  <arg name="left_cam_segmentation_topic" default="/tesse/segmentation/image_raw"/>
  <arg name="left_cam_depth_topic"        default="/tesse/depth/image_raw"/>
  <arg name="transform"                   default="/transform"/>
  <arg name="use_freespace_pointcloud"    default="false" />
  <arg name="freespace_pointcloud"        default="/dev/null"/>

  <!-- Run rosbag if requested with play_bag arg -->
  <node name="player" pkg="rosbag" type="play" output="screen"
    args="--clock --rate $(arg rosbag_rate) $(arg bag_file)"  if="$(arg play_bag)">
    <!-- The rosbag we first generated did not follow ROS naming standards for image topics,
         so we remap the topics accordingly here.-->
    <remap from="/rgb_image_raw"     to="$(arg left_cam_topic)"/>
    <remap from="/depth_image_raw"   to="$(arg left_cam_depth_topic)"/>
    <remap from="/labels_image_raw"  to="$(arg left_cam_segmentation_topic)"/>
    <remap from="/orb_slam_pose"     to="$(arg transform)"/>
  </node>

  <!-- Generate input pointcloud with semantic labels for kimera-semantics:
     - 1. Using the depth image and registered semantic image (run_stereo_dense=false).
     - 2. Using stereo depth reconstruction (run_stereo_dense=true). -->
  <arg name="publish_point_clouds" default="true"/>
  <arg name="run_stereo_dense"     default="false"/>
  <group if="$(arg publish_point_clouds)">
    <!-- Launch Nodelet manager: used by depth_image_proc and disparity_image_proc -->
    <node pkg="nodelet" type="nodelet" name="nodelet_manager" args="manager"
      output="screen"/>
    <!-- Run stereo_dense_reconstruction_node (only if we don't use the depth image). -->
    <arg name="left_cam_stereo_depth_topic" value="/depth_image"/>
    <group if="$(arg run_stereo_dense)">
      <arg name="pointcloud" value="/points2"/>
      <include file="$(find kimera_semantics_ros)/launch/stereo_depth.launch">
<arg name="left_cam_info_topic"         value="$(arg left_cam_info_topic)"/>
        <arg name="right_cam_info_topic"        value="$(arg right_cam_info_topic)"/>
        <arg name="left_cam_topic"              value="$(arg left_cam_topic)"/>
        <arg name="right_cam_topic"             value="$(arg right_cam_topic)"/>
        <arg name="pointcloud"                  value="$(arg pointcloud)"/>
        <arg name="disparity_image"             value="/disparity"/>
        <arg name="depth_image"                 value="$(arg left_cam_stereo_depth_topic)"/>
        <arg name="convert_disparity_img_to_depth_img" value="true"/>
      </include>
    </group>

    <!-- Converts registered depth image and RGB image into an RGB pointcloud.
         Using depth and semantic image, it generates semantic pointcloud. -->
    <node pkg="nodelet" type="nodelet" name="cloudify"
      args="load depth_image_proc/point_cloud_xyzrgb nodelet_manager
      -no-bond" output="screen">
      <!-- Input -->
      <remap from="/rgb_camera_info"            to="$(arg left_cam_info_topic)"/>
      <remap from="/labels_image_raw"           to="$(arg left_cam_segmentation_topic)" if="$(arg metric_semantic_reconstruction)"/>
      <remap from="/rgb_image_raw"              to="$(arg left_cam_topic)"              unless="$(arg metric_semantic_reconstruction)"/>
      <remap from="depth_registered/image_rect" to="$(arg left_cam_stereo_depth_topic)" if="$(arg run_stereo_dense)"/>
      <remap from="/depth_image_raw"            to="$(arg left_cam_depth_topic)"        unless="$(arg run_stereo_dense)"/>
      <remap from="/orb_slam_pose"              to="$(arg transform)"/>
      <!-- Output -->
      <remap from="depth_registered/points"     to="$(arg semantic_pointcloud)"/>
      <!-- Params -->
      <param name="queue_size" value="20"/>
    </node>
  </group>

  <arg name="pkg_type"    default="kimera_semantics_ros"  if="$(arg metric_semantic_reconstruction)"/>
  <arg name="server_type" default="kimera_semantics_node" if="$(arg metric_semantic_reconstruction)"/>
  <arg name="pkg_type"    default="voxblox_ros" unless="$(arg metric_semantic_reconstruction)"/>
  <arg name="server_type" default="tsdf_server" unless="$(arg metric_semantic_reconstruction)"/>
  <node name="kimera_semantics_node" pkg="$(arg pkg_type)" type="$(arg server_type)" output="screen"
    args="-alsologtostderr -colorlogtostderr" clear_params="true">
    <!-- Input -->
    <remap from="pointcloud"                to="$(arg semantic_pointcloud)"/>
    <remap from="orb_slam_pose"             to="$(arg transform)" />
    <!-- Params -->
    <param name="tsdf_voxel_size"           value="$(arg voxel_size)" />
    <param name="tsdf_voxels_per_side"      value="32" />
    <param name="max_ray_length_m"          value="$(arg max_ray_length_m)" />
    <param name="min_time_between_msgs_sec" value="0.2" />
    <param name="voxel_carving_enabled"     value="true" />
    <param name="color_mode"                value="lambert_color"/>
    <param name="use_const_weight"          value="false" />
    <param name="use_freespace_pointcloud"  value="$(arg use_freespace_pointcloud)" />
    <remap from="freespace_pointcloud"      to="$(arg freespace_pointcloud)"/>

    <param name="sensor_frame"              value="$(arg sensor_frame)"/>
    <param name="use_tf_transforms"         value="false" />

    <param name="enable_icp"                value="false" />
    <param name="icp_iterations"            value="10" />

    <param name="verbose"                   value="true" />

I see the mesh generation happening but 3D reconstruction is not shown on rviz. Any pointers on this front will really helpful

from voxblox.

Getting the "Using latest TF transform instead of timestamp match." warning and not sure why... about voxblox HOT 12 CLOSED

Comments (12)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent