Performance Issues about rosbag2 HOT 15 CLOSED

The SQLite plugin is not optimized as of now - rosbag2 still lacks a good performance testing suite to check for bottlenecks. Since no transactions are used as of today in the database backend, I would expect performance to be really bad for many small messages and better for larger messages at lower numbers.

Sending around 100 MB of data per second could be too much already on some systems.

from rosbag2.

@MCP-TTC we indeed fixed a memory leak recently in #502. It's yet to be backported and released in the next Foxy patch release.
As of now, you could try to compile rosbag2 in an overlay workspace with this PR: #518

from rosbag2.

@Karsten1987 I confirm you it works, no more memory leaks. Thanks.

I have a question for you and the community ofc, still about performances, if I record the following topics:

• image1 (like the above) 20Hz
• pcl2 100Hz
• imu 125Hz
• image2 (1/3 of image1 payload) 30Hz

in a single bag instance I notice a big drop of the small but high frequency messages (delta freq. > 40%).
in multiple bag instances (1 per topic for example) the recorded messages frequencies are almost the produced ones (delta freq. = ~4%).

Did anyone ever notice something like this?

PS: the SSD drive can reach the required write bandwidth

from rosbag2.

I'm going to close this issue for now, please reopen if there is anything specific to address.

from rosbag2.

Hi,
have you managed to solve this?
I am facing the same issue on ros galactic version. I use the python rosbag tool to convert the euroc mav dataset from ros1 to ros2. And while in ros1 I get that the imu topic is at 200Hz, at ros2 I get that the imu topic is at 120Hz.

from rosbag2.

Hmm, looks like we have to dig a bit deeper into performance optimizations for SQLite3. The assumption that it's more suited for larger data seems to be in conflict with this ros-answers question: https://answers.ros.org/question/328814/ros2-bag-issue/

@Martin-Idel Do you know of any good starting point for diving into these performance topics?

from rosbag2.

I wanted to come back to this issue, because I find it interesting. I think the first step would be to create a good performance test suite to be sure that any changes actually help.

My starting point would probably be https://stackoverflow.com/questions/1711631/improve-insert-per-second-performance-of-sqlite - as you can see there are quite some points to get started. From what I know the actual performance tuning will probably start with:

• Use transactions (this will definitely help, and it might help a lot)
• Use batching within transactions (larger changes required here, but this will help for small messages)
• Look into journal modes (might also require more user settings)

from rosbag2.

Hi,

I am facing a similar problem with the following configuration:

• 2xImage msg @~11FPS 1928x1208x4
• 1xPointCloud2 msg @~11FPS 900pts
• sqlite plugin
• Ubuntu 18.04
• USB3.0 External SSD (tested up to 480MB/s)
• default DDS

Here some results:

• ROS2 nodes + rosbag2 from ROS2
  • topic_img_1 - 4.707FPS
  • topic_img_2 - 4.709FPS
  • topic_pts - 4.689FPS
• ROS2 nodes + ros1_bridge + rosbag from ROS1
  • topic_img_1 - 1.524FPS
  • topic_img_2 - 3.352FPS
  • topic_pts - 2.972FPS
• ROS1 nodes + rosbagv2 from ROS1
  • topic_img_1 - 11.256FPS
  • topic_img_2 - 11.250FPS
  • topic_pts - 11.116FPS

Any idea on how to improve rosbag2 performances?
Did anyone find a solution, even a temporary one, for this issue?

from rosbag2.

I tried the following setup with Eloquent and Foxy on Ubuntu 18.04, both built from source:

• publish 1920x600 images at 20 FPS with image_tools::cam2image (burger mode)
  ros2 run image_tools cam2image --ros-args -p burger_mode:=true -p width:=1920 -p height:=600 -p frequency:=20.0
• record the topic /image splitting the bag every 2GB
  ros2 bag record -o test-bag -b 2000000000 /image

With Eloquent everything seems fine but with Foxy the recording process continuously allocates memory up to a memory overflow if not stopped. I am wondering if there is any memory leak or if anyone else is getting the same issue.

from rosbag2.

That behavior is probably related to #436 - the queue can fill up and drop messages while rosbag2 is writing

from rosbag2.

@Karsten1987 @adamdbrw With the recent work done for performance in rosbag2, do you think we should close out this issue?

from rosbag2.

I think so, this was one of the background issues for our work and things improved a lot

from rosbag2.

I am OK with closing this but I had no way to test the relative improvements yet. Have those been already merged to the Foxy branch? To me it seems the new improvements are only available on the Rolling one.
Am I wrong?

from rosbag2.

The improvements are merged into the master branch (which is available as a Rolling package). It is my opinion that doing the feature development on the latest branch meets the requirements of an enhancement ticket - there is no given expectation that a new feature is backported to all live distributions. If a backport is desired for specific features, that can be tracked via a separate follow-up ticket.

For discussion about backporting latest features to Foxy, also see https://discourse.ros.org/t/fast-forward-merging-rosbag2-master-api-to-foxy/18927

from rosbag2.

I have a pointcloud2 dataset recorded with ros1. I converted it to ros2 bag using the rosbags tool. When I try to play the bag file, the RAM consumption ramps up and my system starts lagging. I tried to read the bag file with a custom python script using rosbag2_py and things seem to work without any issues. There is no memory outage. Can anyone please let me know whether this is a known issue with ros2 bag?

from rosbag2.