Really appreciate your work on the Surfel-Fusion. It is not really an issue but is there a possible way that i can actually turn the plain/boring single colored pointcloud into something nicer like endowing the true color of the object to that point?

I run vins-mono (not vins-fusion) with mynteye camera, and set estimate_extrinsic to 2 to calibrate the camera and imu. When I run roslaunch surfel_fusion fuse_depthnet.launch, the information is displayed as follows.
process[surfel_fusion-1]: started with pid [13628] Have the following settings: camera matrix: 358.474 0 388.407 0 359.527 254.769 0 0 1 fuse the distence between 0.500000 m and 3.000000 m. loop_path_input size: =30 poses_database size: =0 loop_path_input size: =31 poses_database size: =0 loop_path_input size: =32 poses_database size: =2 loop_path_input size: =33 poses_database size: =3 loop_path_input size: =34 poses_database size: =4
But when I run rostopic pub / surfel_cmd std_msgs/Int16 "data: 2" is used to end the map creation, the error shown below will appear.
loop_path_input size: =234 poses_database size: =204 [ WARN] [1596005843.725373460]: Surfel STOP! saving pointcloud ... terminate called after throwing an instance of 'pcl::IOException' what(): : [pcl::PCDWriter::writeASCII] Input point cloud has no data! [surfel_fusion-1] process has died [pid 13628, exit code -6, cmd /home/jgx/catkin_surfelmapping_vins/devel/lib/surfel_fusion/surfel_fusion ~grey_image:=/mynteye/left/image_raw ~depth:=/mynteye/depth/image_raw ~loop_path:=/pose_graph/pose_graph_path ~extrinsic_pose:=/vins_estimator/extrinsic __name:=surfel_fusion __log:=/home/jgx/.ros/log/5ea3157c-d168-11ea-b86a-983f9f190824/surfel_fusion-1.log]. log file: /home/jgx/.ros/log/5ea3157c-d168-11ea-b86a-983f9f190824/surfel_fusion-1*.log all processes on machine have died, roslaunch will exit shutting down processing monitor... ... shutting down processing monitor complete done
My camera only has grayscale images and no color images. I don't know if this is the cause.could you help me solve this issue,thans very much.

您好，运行kitti数据集时没有问题，换做自己的realsense d435i相机后，运行surfel map时，程序没有继续执行，一直显示：
begin new frame process!!!
warp the surfels according to the loop!
warp end! cost 0.000224 ms.

Hello, Dr. Wang, I saw on the author's Github of MMGCN that you reproduced the paper and raised some questions to the author at that time. Do you still have the data set of that paper?Can you share it with me?

Thank for your nice work!
I would like to reproduce the result on ICL-NUIM dataset of DSM in the paper.
I choose sequence living_room_traj0n_frei as test sequence. ORB-SLAM2 works well in RGBD mode, but surfel_fusion died.
the launch file is as follows:

After tracking the errors, I found that the position where the problem occured:

When fusing surfels, the variable sp_index sometimes is -1 which is out of range.
I'd like to know whether if it is normal that the superpixel index is -1 after superpixel segmentation in DSM. Or there exists some errors in my parameter setting?

I changed the camera parameters and topic name to realsense in kitti_orb.launch. What I changed are as follows:

<launch>
  <node pkg="surfel_fusion" type="surfel_fusion" name="surfel_fusion" clear_params="true" output="screen">

    <!-- camera parameter -->
    <param name="cam_width" value="640" />
    <param name="cam_height" value="480" />

    <!--input image info-->
    <param name="cam_fx" value="614.4114379882812" />
    <param name="cam_cx" value="324.2138671875" />
    <param name="cam_fy" value="614.7125244140625" />
    <param name="cam_cy" value="236.86329650878906" />

    <!-- fusion parameter, all in meter -->
    <param name="fuse_far_distence" value="30.0" />
    <param name="fuse_near_distence" value="0.5" />

    <!-- for deform the map -->
    <param name="drift_free_poses" value="30" />

    <!-- for data save -->
    <!-- <param name="save_name" value="/home/wangxinxin/bag/surfel_fusion/results/rgbd/realsense_loop" /> -->

    <remap from="~image" to="/camera/color/image_raw" />
    <remap from="~depth" to="/camera/depth/image_rect_raw" />
    <remap from="~loop_path" to="/orb_slam/path" />
    <remap from="~this_pose" to="/orb_slam/pose" />
    <remap from="~loop_stamps" to="/orb_slam/loop" />
  </node>
</launch>

But surfel fusion can not work as follow:

When I use kitti which you used, it will be ok. Such as follow:

I don't kown why the phenomenon will occure, can you help me?

Hello, rviz always crashes when I load rviz_config.rviz in the last step. Why is this?

你好，我用intel realsense D435i跑orbslam2的rgbd模式，建立地图的时候，当orbslam2 track lost时， track_result为空，造成段错误（核心转储）错误，这怎么解决？保存点云文件时local_surfe的update_times都为1，更新次数小于5， 没有保存点云信息是我哪有啥错误吗？期待你的回应，谢谢！

Hello, recently read your article, and very interested in your research, but I encountered some difficulties in installing your code, need your help!
Now I have installed orb according to your steps_ Slam2, but according to your tips, I don't know how to install "Surfel_fusion" and "Kitti publisher", can you give me some more detailed tips or can someone else help me? I will thank you very much.

Nothing happens after opening the provided rviz file.
Wondering what to adjust in addition to receiving topics if manually adjusted.

您好,我运行这个程序的时候,弹出来的小窗里会显示出surfel,但是我不想让他跟随我的镜头,我该怎么将他固定在原点呢?就是我想看全貌图

Hi,
I see the code remap from="~depth" to="/camera/aligned_depth_to_color/image_raw in vins_realsense.launch file,
so does this mean the depth information is directly come from the camera sensor, not computed by adjacent images?

Could anybody share the dataset link again,thanks .
https://www.dropbox.com/s/qpn40yt8bjvkapd/kitti_sequence_00.tar.gz?dl=0
can't download

Now being the paper been accepted, will it be possible for you to share the source code?

I have tested the DenseSurfelMapping. I think it is very good. I have read this sentence " For VINS-MONO, we are using it in a teach-and-repeat project and will opensource lately". Do you have opened the source of VINS-MONO for DenseSurfelMapping for open source community?
Thank you very much

Hi,thank you for your work! However,the code of vins-supported seem can't save the map.What do I need to do to save the map?

Thank you!

Hi,

Could you explain if MVSDepthNet was running in real-time to generate the depth maps? If so, was it running on the drone?

Thanks! :)

hi, Mr.wang. would you please give me some advice about why The map is so sparse when VO provided by VINS-FUSION compare with ORB-SLAM. As shown follow:（white:VINS-fusion for VO&&"DenseSurfelMapping-VINS-supported" for mapping. color:ORB-SLAM for VO&&"DenseSurfelMapping-master" for mapping）looking forward for your reply! Thank u!

hi thanks for releasing your wonderful work! the depth map come from PSMnet as mentioned in introduction page. I wonder have you ever post-processed the depth of PSMnet such as re-scale or you just directly take the output of PSMnet as depth.

Your work is great and I would like to learn it. But when downloading the KITTI dataset you gave it shows that the link does not exist. Is this dataset different from the KITTI official website? There are only stereo images on the official website. The KITTI connection you gave should contain depth values, right? Could you please send me a copy, thanks!

Hello, when I catkin_make kitti_publisher, the following error occurred
Could not find a package configuration file provided by "depthnet_msgs"
   With any of the following names:

     depthnet_msgsConfig.cmake
     Depthnet_msgs-config.cmake
What is going on with this?

有没有有同感的朋友，或者我设置有错误？

Dear author @WANG-KX,

Thanks for your this open-source code. Great work!
I noticed that in your ORB-supported implementation, map could be built without revisiting.

However, in your Vins-supported implementation, map could be built only after having some revisiting. If no revisiting is done, nothing will be saved in the map.

I am wondering what could be the reason and is there any possibility to save the map in Vins-supported without revisiting (of course revisiting could eliminate the mapping error)? Can you give me some hints on this point? Thanks!

Hi, when I try this code on my own device (d435i), I met this problem. The depth msg cannot be received by surfel_fusion. I think the sync policy has some bug. Thanks very much.

你好，我想问一下，Real-time Scalable Dense Surfel Mapping这篇文章在哪儿能下载到？

感谢开源这么好的DenseSurfelMapping，在fusion_functions.cpp第208行
if (reference_frame_index - local_surfels[i].last_update >10 && local_surfels[i].update_times < 6)
其中10应为从launch文件中读取的drift_free_poses，不然保存的mesh有问题，谢谢！

README.md specifies that this project is "open source". Is there a license that can be added to the repository?

Thanks for sharing the mapping tool!
While I ran the Kitti00 dataset, it seemed like that the surfel-mapping thread worked normally, and I could observe colorful surfels in the debug-show window as shown in Fig 1. But when I turned on the Rviz launch file, some troubles was met.
Firstly the rivz file can not be read properly as I ran "rosrun rviz rviz path-to-rviz-file" in Fig 2 and 3, and the pointclouds were all colorful whether active or inactive (which should be gray as I watched other issues or the youtube video uploaded), finally the view point did not follow the camera motion. As I was not familar with the rviz options, would you plz share the right way to launch rviz?

Thank you for your work，and release it. When i use ORB_slam2 model RGBd
to run Kitti dataset, sequence 00, there is no problem. And i can get the ideal result. But i processing our outdoor road data, it don't extract the super pixels, and don't generate the point cloud. I don't know which problem it is. Can you help me to solve it?　I am looking forward your reply!!! Thank you!

Thank you for your excellent work. When I use DenseSurfelMapping, I use VINS-Fusion, and the camera uses Realsense D435i, but when I use it, DenseSurfelMapping does not work. There is no information when I open spImage in rviz. Do I need to synchronize the posture information with the image and depth map in the DenseSurfelMapping code before fusing it? If so, how should it be synchronized?
Thanks again for your work!

when I running
roslaunch surfel_fusion vins_realsense.launch

and mapping 3d map
after a time while,
the terminal shows 👍

terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
[surfel_fusion-1] process has died [pid 24384, exit code -6, cmd /home/dell/denseSurfel_vino_supptorted_src/devel/lib/surfel_fusion/surfel_fusion ~image:=/camera/color/image_raw ~depth:=/cam

具体测试截图如下：


生成的深度图如下：

求高手解答，谢谢！

The work is excellent! Can you give some guides about run with RDB-D data such as TUM or ICL-NIUM?

The reconstruction rsult is quiet uneven.As shown after,a small part of the provided dataset is much better than others.

I'm curious why.

当我单独运行vins_mono的时候vins的效果是正常的，但是当我把vins_mono和surfelMapping同时开启的时候效果非常的差。vins几乎失效了，vins很难进行初始化，即使初始化成功后很快也会失效需要重新初始化。两者同时开启后vins计算不出我行走的轨迹，rviz中的摄像头模型基本就在原点附近飘动。

一开始我怀疑是我的电脑配置的问题，我把vins和surfelMapping分开到两台电脑中运行。一台电脑单独运行vins的时候是没问题的，当另一台电脑开启了surfelMapping订阅vins发出的话题时效果和我前面说的一样vins的效果变得了很差，几乎无法使用了。

Hi,

When I try to run

ROS_PACKAGE_PATH="${ROS_PACKAGE_PATH}:$(pwd)/surfel_fusion" \
 roslaunch surfel_fusion kitti_orb.launch

Some of it starts, but not all:

...
Checking log directory for disk usage. This may take a while.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://localhost:45161/

SUMMARY
========

CLEAR PARAMETERS
 * /surfel_fusion/

PARAMETERS
 * /rosdistro: melodic
 * /rosversion: 1.14.4
 * /surfel_fusion/cam_cx: 607.1928
 * /surfel_fusion/cam_cy: 185.2157
 * /surfel_fusion/cam_fx: 718.856
 * /surfel_fusion/cam_fy: 718.856
 * /surfel_fusion/cam_height: 376
 * /surfel_fusion/cam_width: 1241
 * /surfel_fusion/drift_free_poses: 10
 * /surfel_fusion/fuse_far_distence: 30.0
 * /surfel_fusion/fuse_near_distence: 0.5

NODES
  /
    surfel_fusion (surfel_fusion/surfel_fusion)

auto-starting new master
process[master]: started with pid [400]
ROS_MASTER_URI=http://localhost:11311

setting /run_id to d5f8dab8-6ec0-11ea-a215-0242c0a80002
process[rosout-1]: started with pid [411]
started core service [/rosout]
ERROR: cannot launch node of type [surfel_fusion/surfel_fusion]: Cannot locate node of type [surfel_fusion] in package [surfel_fusion]. Make sure file exists in package path and permission is set to executable (chmod +x)

any hints on how to make it work?

Thanks a lot,
Aron

After running the code, i encountered two problems in Surfel_fusion。The first one is process has died [pid xxx, exit code -11, cmd ...]. Another is Surfel_fusion sometimes can not receive the pose of ORB_SLAM2. Thanks for the reply!

When I run this code on my computer, the generated ply file cannot be opened normally, cloudcompare and meshlab are not working, how can I solve this problem?

Error in CloudCompare:
[PLY]'Unexpected end of File'
[PLY]'Error reading 'x' of 'vertex' number '1919032'
An error occurred while loading 'resule_00_mesh': the thirdparty library in charge of saving/loading the file has failed to perform the operation.

Error in MeshLab:
Error : Unespected eof

Hi,
This really is very amazing work，it's practical in engineering.
en, I want to know the evaluation tool of Map accuracy and Visualization Tools，to test my own data.

Thanks for your work!
I try to use the VIN-supported branch but I met some problems.
The following picture is just used to show what my dataset looks like

There is a table and there is some stuff on that table, the camera will move around the table and keep filming the table. I publish the raw point cloud to rviz. The topic is surfel_fusion/raw_pointcloud as shown in your code。

You can see the desk is moving. But the problem is that the desk should not move. The camera is moving but the scene should standstill.
If I don't publish the raw point cloud and publish the point cloud(pointcloud_noceil) instead I'll end up with a point cloud looks like the following in rviz

Sorry maybe you cannot see it very clearly but you focus on the moving part of the point cloud you can still see there is a round table that is moving.
What I hope is the table remain still, the scene may extend as the camera moves. Now the table seems to move along with the camera.

The rosbag's topic is

topics:    /depth         1169 msgs    : sensor_msgs/Image        
             /groundtruth   3003 msgs    : geometry_msgs/PoseStamped
             /image0        1169 msgs    : sensor_msgs/Image             
             /imu0          9069 msgs    : sensor_msgs/Imu

The depth is strictly align with the image0.
I modify your surfel_fusion launch file to (ignore the camera intrinsics setting)

...
    <remap from="~image" to="/image0" />
    <remap from="~depth" to="/depth" />
   <remap from="~loop_path" to="/vins_estimator/path" />
   <remap from="~extrinsic_pose" to="/vins_estimator/extrinsic" />

I play the rosbag, VINS-Mono will subscribe to image0, imu0 to generate the camera's motion. Your package will subscribe to image0, depth directly from rosbag and /vins_estimator/path, /vins_estimator/extrinsic generated by VINS. Correct me if my operation is wrong.
The VINS itself can give me pretty good pose estimation. I have compared it with the ground truth.

Hello, can you provide the baidu cloud disk link of the dataset sequence 00?
I can not download the data from the link you provided.

您好
请问能不能直接根据已有的位姿，以及RGBD图像来建图呢？
如果能，那我需要对哪里进行修改呢？

I want to run program on an indoor dataset to observe the result of the reconstruction. I modify the camera Intrinsic in the launch file for node surfer_fusion, and rewrite the node that sends the picture like this:
depth = cv::imread(PATH+s1, -1);
left = cv::imread(PATH+s2,-1);
cvtColor( left, left, CV_BGR2GRAY );
depth.convertTo(depth,CV_32F);
sensor_msgs::ImagePtr msg_1 = cv_bridge::CvImage(std_msgs::Header(), "mono8", left).toImageMsg();
sensor_msgs::ImagePtr msg_2 = cv_bridge::CvImage(std_msgs::Header(), "32FC1", depth).toImageMsg();
ORB-SLAM2 runs well as usual,but surfer_fusion cannot create any surfel,is there other part of the program that should be modified?

This work seems great. I'm wondering if I can use it in an offline mode, with just RGBD data, and known camera poses.

Thanks

Hi,
Thanks for the project, I want to ask if I can use sgm from https://github.com/fixstars/libSGM to calculate disparity maps or there is certain methods that should be used

Hi, Wang, I found that there is a transform dedicated for kitti. Is it needed for every SLAM system? Can I delete it?

static Eigen::Matrix4d transform_kitti;
{
Eigen::Matrix4d received_pose;
pose_ros2eigen(input_pose, received_pose);
if(poses_database.size() == 0)
{
Eigen::Matrix4d idea_pose;
idea_pose = Eigen::Matrix4d::Zero();
idea_pose(0,0) = 1.0;
idea_pose(1,2) = 1.0;
idea_pose(2,1) = -1.0;
idea_pose(3,3) = 1.0;
transform_kitti = idea_pose * received_pose.inverse();
}
Eigen::Matrix4d transformed_pose;
transformed_pose = transform_kitti * received_pose;
pose_eigen2ros(transformed_pose, input_pose);
}
transform end