hridaybavle / semantic_slam Goto Github PK

This package is for running the semantic SLAM algorithm using extracted planar surfaces from the received detection

CMake 6.84% C++ 93.00% Shell 0.16%

semantic_slam's Introduction

Semantic SLAM

This package can perform optimization of pose estimated from VO/VIO methods which tend to drift over time. It uses planar surfaces extracted from object detections in order to create a sparse semantic map of the environment, thus optimizing the drift of the VO/VIO algorithms.

In order to run this package you will need two additional modules

A VO/VIO algorithm: ROVIO, VINS_MONO, OKVIS etc.
An Object Detector: Yolo, Shape Color Detector

Currently it can extract planar surfaces and create a semantic map from from the following objects:

chair
tvmonitor
book
keyboard
laptop
bucket
car

Related Paper:

@ARTICLE{9045978,
  author={Bavle, Hriday and De La Puente, Paloma and How, Jonathan P. and Campoy, Pascual},
  journal={IEEE Access}, 
  title={VPS-SLAM: Visual Planar Semantic SLAM for Aerial Robotic Systems}, 
  year={2020},
  volume={8},
  number={},
  pages={60704-60718},
  doi={10.1109/ACCESS.2020.2983121}}

Video

How do I set it up?

First install g2o following these instructions (Tested on Kinetic and Melodic Distributions):

- sudo apt-get install ros-$ROS_DISTRO-libg2o
- sudo cp -r /opt/ros/$ROS_DISTRO/lib/libg2o_* /usr/local/lib
- sudo cp -r /opt/ros/$ROS_DISTRO/include/g2o /usr/local/include

Install OctopMap server for map generation capabilities:

- sudo apt install ros-$ROS_DISTRO-octomap*

Try a simple example with pre-recorded VIO pose and a blue bucket detector:

Create a ros workspace and clone the following packages:

Download the rosbag:

    wget -P ~/Downloads/ https://www.dropbox.com/s/jnywuvcn2m9ubu2/entire_lab_3_rounds.bag

Create a workspace, clone the repo and compile:

    mkdir -p workspace/ros/semantic_slam_ws/src/ && cd workspace/ros/semantic_slam_ws/src/    
    git clone https://github.com/hridaybavle/semantic_slam && git clone https://bitbucket.org/hridaybavle/bucket_detector.git   
    cd .. && catkin build --cmake-args -DCMAKE_BUILD_TYPE=Release

Launch and visualize

    source devel/setup.bash
    roslaunch semantic_SLAM ps_slam_with_snap_pose_bucket_det_lab_data_with_octomap.launch bagfile:=${HOME}/Downloads/entire_lab_3_rounds.bag show_rviz:=true

Using Docker Image

If the code is giving problems with you local machine, you can try the docker image created with the repo and the required settings.

Download Docker from: Docker

Follow the commands to run the algorithm with the docker

  docker pull hridaybavle/semantic_slam:v1 	
  docker run --rm -it --net="host" -p 11311:11311 hridaybavle/semantic_slam:v1 bash
  cd ~/workspace/ros/semantic_slam_ws/
  source devel/setup.bash
  roslaunch semantic_SLAM ps_slam_with_snap_pose_bucket_det_lab_data_with_octomap.launch bagfile:=${HOME}/Downloads/entire_lab_3_rounds.bag show_rviz:=false

Open a new terminal and rviz in local machine

  cd ~/Downloads/ && wget https://raw.githubusercontent.com/hridaybavle/semantic_slam/master/rviz/graph_semantic_slam.rviz
  rviz -d graph_semantic_slam.rviz

Subsribed Topics

/SQ04/snap_vislam/vislam/pose (geometry_msgs/PoseStamped)
The default snapdragon VIO pose published in NED in frame. This message can be remapped remapped to any other VO pose message publishing in NED frame. (See frame conventions)
/rovio/odometry (geometry_msgs/PoseStamped)
The VIO odometry published in ENU frame. Can be remapped to the desired topic name in the launcher.
/depth_registered/points (sensor_msgs/PointCloud2) The point cloud required for planar surface extraction.
/darknet_ros/bounding_boxes(darknet_msgs_ros/BoundingBoxes)
The detection bounding boxes published by yolo if using the yolo detector ros package.
/image_processed/bounding_boxes(ShapeColor_ObjectDetection/DetectedObjects) The detection bounding boxes if using the bucket detector. It can be downloaded from the link above.

Published Topics

robot_pose(geometry_msgs/PoseStamped) The pose of the robot estimated by the algo.
robot_path(nav_msgs/Path)
The path of the robot psoe for visualization.
keyframe_poses(geometry_msgs/PoseArray)
The poses of the keyframe being added to the g2o graph.
mapped_landmarks(visualization_msgs/MarkerArray)
The mapped semantic planar surfaces.
detected_landmarks(visualization_msgs/MarkerArray)
The detected landmarks in the current frame.

The configurations of the algorithms can be found inside the cfg folder in order to be changed accordingly.

Published TFs

map to odom transform: The transform published between the map frame and the odom frame after the corrections from the semantic SLAM.
base_link to odom transform: The transform published between the base_link (on the robot) frame and the odom frame as estimated by the VO/VIO algorithm.

semantic_slam's People

Contributors

Stargazers

Watchers

semantic_slam's Issues

license ?

errors at last step

Hi, I have finished all the steps following the instructions and nothing goes wrong.
But when I run

roslaunch semantic_SLAM ps_slam_with_snap_pose_bucket_det_lab_data.launch bagfile:=${HOME}/Downloads/entire_lab_3_rounds.bag show_rviz:=true

I get errors like this and it stucks for a while.

# Using CSparse poseDim -1 landMarkDim -1 blockordering 0
done
keyframe_delta_trans 0.5
keyframe_delta_angle 0.5
keyframe_delta_time 1
use_const_inf_matrix: 1
const_stddev_x: 0.00667
const_stddev_q: 1e-05
Initialized mapping thread 
camera angle in radians: 0.59219
update keyframe every detection: 1
add first landmark: 0
[semantic_graph_slam_node-9] process has died [pid 23067, exit code -11, cmd /home/nrc/workspace/ros/semantic_slam_ws/devel/lib/semantic_SLAM/semantic_graph_SLAM_node __name:=semantic_graph_slam_node __log:=/home/nrc/.ros/log/ccaf4b14-a47a-11ea-b300-000c29c39525/semantic_graph_slam_node-9.log].
log file: /home/nrc/.ros/log/ccaf4b14-a47a-11ea-b300-000c29c39525/semantic_graph_slam_node-9*.log

then I get this. It seems that the visualization program doesn't go right.

[rosbag-2] process has finished cleanly
log file: /home/nrc/.ros/log/ccaf4b14-a47a-11ea-b300-000c29c39525/rosbag-2*.log

Is there something I have missed?
Thank you!

Dataset download failure problem

Hello, when I run the sample code, I always encounter network interruptions or unknown errors at the last moment when downloading the dataset entire_lab_3_rounds.bag. Can you provide a new way to download the bag?

[semantic_graph_slam_node-9] process has died

Hi, I have finished all the steps following the instructions and nothing goes wrong.
But when I run

roslaunch semantic_SLAM ps_slam_with_snap_pose_bucket_det_lab_data_with_octomap.launch bagfile:=${HOME}/Downloads/entire_lab_3_rounds.bag show_rviz:=true

I get errors like this.

done
keyframe_delta_trans 0.5
keyframe_delta_angle 0.5
keyframe_delta_time 1
use_const_inf_matrix: 1
const_stddev_x: 0.00667
const_stddev_q: 1e-05
camera angle in radians: 0.59219
update keyframe every detection: 1
add first landmark: 0
[ INFO] [1591944956.099907360, 1661396775.076756992]: waitForService: Service [/depth_rectifier_manager/load_nodelet] is now available.
[ INFO] [1591944956.100243666, 1661396775.076756992]: waitForService: Service [/depth_manager/load_nodelet] is now available.
[ INFO] [1591944956.545617511, 1661396775.518832629]: Stereo is NOT SUPPORTED
[ INFO] [1591944956.545842654, 1661396775.518832629]: OpenGl version: 4.5 (GLSL 4.5).
[pcl::OrganizedNeighbor::radiusSearch] Input dataset is not from a projective device!
Residual (MSE) 0.000614, using 1248 valid points
[pcl::OrganizedNeighbor::radiusSearch] Input dataset is not from a projective device!
Residual (MSE) 0.000748, using 1444 valid points
[pcl::OrganizedNeighbor::radiusSearch] Input dataset is not from a projective device!
Residual (MSE) 0.001710, using 2303 valid points
[semantic_graph_slam_node-9] process has died [pid 27314, exit code -9, cmd /home/nrc/hd/workspace/ros/semantic_slam_ws/devel/lib/semantic_SLAM/semantic_graph_SLAM_node __name:=semantic_graph_slam_node __log:=/home/nrc/.ros/log/c2c4ddd8-ac79-11ea-96ed-8ca982ff1833/semantic_graph_slam_node-9.log].
log file: /home/nrc/.ros/log/c2c4ddd8-ac79-11ea-96ed-8ca982ff1833/semantic_graph_slam_node-9*.log

When it occurs

[pcl::OrganizedNeighbor::radiusSearch] Input dataset is not from a projective device!
Residual (MSE) 0.000614, using 1248 valid points

the program is still mapping , so I think the problem is not caused by 'pcl'.

I tried to run the launchfile seperately :

ROS_NAMESPACE=camera/color rosrun image_proc image_proc 
roslaunch semantic_SLAM shape.launch  
rosrun semantic_SLAM  semantic_graph_SLAM_node

But at the last step i got 'Segmentation fault :

add first landmark: 0
Segmentation fault (core dumped)

Do you have any idea about it?
Thanks a lot !!!

Problemas de incompatibilidad de opencv en el bucket detector

Hola,
he conseguido que ambos paquetes en conjunto (semantic slam y bucket detector) funciones bien en una distro de ubuntu virgen con ROS melodic, sin embargo, cuando migro al pc donde trabajo habitualmente y que tiene ya instaladas dependencias anteriores y demás me encuentro con estos errores referentes a opencv:

Imagino que se deben a incompatibilidades entre versiones de opencv, podrías confirmarme esto último? Sería posible trabajar con una versión de opencv diferente?
Gracias, un saludo!