The first draft of SQLite3 (#6) has to adhere to the abstract plugin API.

When building ros2 from source, the rosbag2 test "test_record_all" apparently does not play nice with tests from other packages because it starts to pick up topics from other tests.

If you do 'colcon test' and other tests are allowed to run in parallel, the rosbag2_transport tests will fail

To reproduce

In a source workspace:

colcon build --packages-up-to rosbag2_transport
source install/setup.sh
while ./build/rosbag2_transport/test_record_all; do :; done

This will run the test in a loop, and it should pass every time. The output from a good test looks something like this:

Running main() from gmock_main.cc
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from RecordIntegrationTestFixture
[ RUN      ] RecordIntegrationTestFixture.published_messages_from_multiple_topics_are_recorded
[INFO] [rosbag2_transport]: Listening for topics...
[INFO] [rosbag2_transport]: Subscribed to topic '/parameter_events'
[INFO] [rosbag2_transport]: Subscribed to topic '/chatter'
[INFO] [rosbag2_transport]: Subscribed to topic '/string_topic'
[INFO] [rosbag2_transport]: Subscribed to topic '/array_topic'
[       OK ] RecordIntegrationTestFixture.published_messages_from_multiple_topics_are_recorded (274 ms)
[----------] 1 test from RecordIntegrationTestFixture (274 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (274 ms total)
[  PASSED  ] 1 test.

Now, in another terminal, do

colcon test --packages-up-to rclcpp

You will probably see the rosbag2 transport tests fail with a message like:

Running main() from gmock_main.cc
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from RecordIntegrationTestFixture
[ RUN      ] RecordIntegrationTestFixture.published_messages_from_multiple_topics_are_recorded
[INFO] [rosbag2_transport]: Listening for topics...
[INFO] [rosbag2_transport]: Subscribed to topic '/parameter_events'
[INFO] [rosbag2_transport]: Subscribed to topic '/chatter'
[INFO] [rosbag2_transport]: Subscribed to topic '/string_topic'
[INFO] [rosbag2_transport]: Subscribed to topic '/array_topic'
[INFO] [rosbag2_transport]: Subscribed to topic '/rcl_test_subscription_nominal_string_chatter'
/workspace/src/rosbag2/rosbag2_transport/test/rosbag2_transport/test_record_all.cpp:44: Failure
Value of: recorded_messages
Expected: size is equal to 4
  Actual: { 16-byte object <50-A4 65-95 AD-7F 00-00 40-A4 65-95 AD-7F 00-00>, 16-byte object <E0-01 66-95 AD-7F 00-00 D0-01 66-95 AD-7F 00-00>,
 16-byte object <40-5A 32-95 AD-7F 00-00 30-5A 32-95 AD-7F 00-00>, 16-byte object <40-2F 45-95 AD-7F 00-00 30-2F 45-95 AD-7F 00-00>, 16-byte ob
ject <C0-1F 45-95 AD-7F 00-00 B0-1F 45-95 AD-7F 00-00> }, whose size 5 doesn't match
[  FAILED  ] RecordIntegrationTestFixture.published_messages_from_multiple_topics_are_recorded (281 ms)
[----------] 1 test from RecordIntegrationTestFixture (281 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (281 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] RecordIntegrationTestFixture.published_messages_from_multiple_topics_are_recorded

 1 FAILED TEST

You can see from the fosbag2_transport INFO log message in the test, that it picked up an extra topic /rcl_test_subscription_nominal_string_chatter This particular example came from here but in our job, I've also seen it conflict with other test topics

3: [INFO] [rosbag2_transport]: Listening for topics...
3: [INFO] [rosbag2_transport]: Subscribed to topic '/string_topic'
3: [INFO] [rosbag2_transport]: Subscribed to topic '/my_ns/parameter_events'
3: [INFO] [rosbag2_transport]: Subscribed to topic '/parameter_events'
3: [INFO] [rosbag2_transport]: Subscribed to topic '/my_ns/chatter'
3: [INFO] [rosbag2_transport]: Subscribed to topic '/chatter_composition'
3: [INFO] [rosbag2_transport]: Subscribed to topic '/array_topic'
3: [INFO] [rosbag2_transport]: Subscribed to topic '/logging_demo_count'

How to fix?

Some options:

• Could we make the test assert that size of the recorded messages is 4 in case some other topics got sampled?
• Is there a way to namespace the topics that the test is recording?
  • Related: can we tell rosbag2 to only record from /array_topic and /string_topic?
• Could we set some DDS isolation for the test?

I can take a crack at this - but wanted to see if anybody had a preferred approach first.

Options are:

ros2 bag record -a --> record all available topics
ros2 bag record topic1 [topic2 ... topicN] --> record specific topics

Connects to #11
With the help of the ROS 1 bridge there shall be a convert function which converts ROS 1 binary data into a ROS 2 message.

Connects to #11
Implement a convert plugin to convert CDR to ROS 2 and vice versa.

connected to ros2/rclcpp#663

original issue post:

I have a custom ROS2 message, which I built and sourced correctly and it successfully publishes and subscribes on the node I create. However, when I try to record the bag files using,

ros2 bag record -topicname

it throws this error,

Failed to record: Something went wrong loading the typesupport library for message type package_name/msgname. Library could not be found.

I don't know why this error, is it because it is a custom message? Or is it because I need to link the custom messages somewhere?

It would be nice to clean up empty bag files if ros2 bag record fails to start or terminates before recording any messages.

Bug report

• Operating System:
  • MacOS 10.14.5 (Mojave)
• Installation type:
  • Binaries
• Version or commit hash:
  • Dashing
• DDS implementation:
  • Fast-RTPS

Steps to reproduce issue

ROS2 bag fails to subscribe to messages generated directly from .idl files. An example package that creates a simple string message using idl can be found at: ros2_idl_demo. Building the message (idl_msgs) and sample publisher (idl_pub) packages and then running ros2 run idl_pub idl_pub and 'ros2 bag record -a' or 'ros2 bag record /chatter' fails to record the message published from idl_pub.

Expected behavior

davidhodo@Davids-MacBook-Pro:idl_test_ws∫ ros2 bag record /chatter
[INFO] [rosbag2_storage]: Opened database 'rosbag2_2019_06_08-12_29_58'.
[INFO] [rosbag2_transport]: Listening for topics...
[INFO] [rosbag2_transport]: Subscribed to topic '/chatter'
[INFO] [rosbag2_transport]: All requested topics are subscribed. Stopping discovery...

Actual behavior

davidhodo@Davids-MacBook-Pro:idl_test_ws∫ ros2 bag record /chatter
[INFO] [rosbag2_storage]: Opened database 'rosbag2_2019_06_08-12_17_47'.
[INFO] [rosbag2_transport]: Listening for topics...

Additional information

Could this be related to the message type / package name changes when using idl directly as described in this issue?

[UPDATE]
In typesupport_helpers.cpp:99 there is a hardcoded "__msg__" in the generation of the symbol name. Changing this to "__idl__" allows the idl message to be recorded at the expense of no longer working for standard messages. This value should be set based on the package type.

Hello,

I have my CI pipeline set up to build rosbag2 on the master release of ROS2. On 4/29 my build failed with the errors below. I started digging into it and it think it might be related to this PR in rcl: ros2/rcl#421

I started digging through it but its many layers of macro's so I am having a hard time pin pointing the exact problem.

inished <<< sqlite3_vendor [2.43s]
Finished <<< rosbag2_test_common [3.64s]
Starting >>> rosbag2_storage
Finished <<< shared_queues_vendor [4.81s]
--- stderr: rosbag2_storage‌
/root/ros2_ws/ros2_ov/rosbag2/rosbag2_storage/src/rosbag2_storage/ros_helper.cpp:54:11: error: format string is not a string literal (potentially insecure) [-Werror,-Wformat-security]
ROSBAG2_STORAGE_LOG_ERROR_STREAM(
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/root/ros2_ws/ros2_ov/rosbag2/rosbag2_storage/include/rosbag2_storage/logging.hpp:40:59: note: expanded from macro 'ROSBAG2_STORAGE_LOG_ERROR_STREAM'
RCUTILS_LOG_ERROR_NAMED(ROSBAG2_STORAGE_PACKAGE_NAME, __ss.str().c_str());
^~~~~~~~~~~~~~~~~~
/root/ros2_ws/ros2_base/install/include/rcutils/logging_macros.h:997:5: note: expanded from macro 'RCUTILS_LOG_ERROR_NAMED'
VA_ARGS)
^~~~~~~~~~~
/root/ros2_ws/ros2_base/install/include/rcutils/logging_macros.h:72:64: note: expanded from macro 'RCUTILS_LOG_COND_NAMED'
rcutils_log(&__rcutils_logging_location, severity, name, VA_ARGS);
^~~~~~~~~~~
/root/ros2_ws/ros2_ov/rosbag2/rosbag2_storage/src/rosbag2_storage/ros_helper.cpp:54:11: note: treat the string as an argument to avoid this
/root/ros2_ws/ros2_ov/rosbag2/rosbag2_storage/include/rosbag2_storage/logging.hpp:40:59: note: expanded from macro 'ROSBAG2_STORAGE_LOG_ERROR_STREAM'
RCUTILS_LOG_ERROR_NAMED(ROSBAG2_STORAGE_PACKAGE_NAME, __ss.str().c_str());
^
1 error generated.
make[2]: *** [CMakeFiles/rosbag2_storage.dir/src/rosbag2_storage/ros_helper.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
In file included from /root/ros2_ws/ros2_ov/rosbag2/rosbag2_storage/src/rosbag2_storage/storage_factory.cpp:24:
/root/ros2_ws/ros2_ov/rosbag2/rosbag2_storage/src/rosbag2_storage/./impl/storage_factory_impl.hpp:92:7: error: format string is not a string literal (potentially insecure) [-Werror,-Wformat-security]
ROSBAG2_STORAGE_LOG_ERROR_STREAM("Unable to create class load instance: " << e.what());
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/root/ros2_ws/ros2_ov/rosbag2/rosbag2_storage/include/rosbag2_storage/logging.hpp:40:59: note: expanded from macro 'ROSBAG2_STORAGE_LOG_ERROR_STREAM'
RCUTILS_LOG_ERROR_NAMED(ROSBAG2_STORAGE_PACKAGE_NAME, __ss.str().c_str());
^~~~~~~~~~~~~~~~~~
/root/ros2_ws/ros2_base/install/include/rcutils/logging_macros.h:997:5: note: expanded from macro 'RCUTILS_LOG_ERROR_NAMED'
VA_ARGS)
^~~~~~~~~~~
/root/ros2_ws/ros2_base/install/include/rcutils/logging_macros.h:72:64: note: expanded from macro 'RCUTILS_LOG_COND_NAMED'
rcutils_log(&__rcutils_logging_location, severity, name, VA_ARGS);
^~~~~~~~~~~
/root/ros2_ws/ros2_ov/rosbag2/rosbag2_storage/src/rosbag2_storage/./impl/storage_factory_impl.hpp:92:7: note: treat the string as an argument to avoid this
/root/ros2_ws/ros2_ov/rosbag2/rosbag2_storage/include/rosbag2_storage/logging.hpp:40:59: note: expanded from macro 'ROSBAG2_STORAGE_LOG_ERROR_STREAM'
RCUTILS_LOG_ERROR_NAMED(ROSBAG2_STORAGE_PACKAGE_NAME, __ss.str().c_str());
^
In file included from /root/ros2_ws/ros2_ov/rosbag2/rosbag2_storage/src/rosbag2_storage/storage_factory.cpp:24:
/root/ros2_ws/ros2_ov/rosbag2/rosbag2_storage/src/rosbag2_storage/./impl/storage_factory_impl.hpp:99:7: error: format string is not a string literal (potentially insecure) [-Werror,-Wformat-security]
ROSBAG2_STORAGE_LOG_ERROR_STREAM("Unable to create class load instance: " << e.what());
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/root/ros2_ws/ros2_ov/rosbag2/rosbag2_storage/include/rosbag2_storage/logging.hpp:40:59: note: expanded from macro 'ROSBAG2_STORAGE_LOG_ERROR_STREAM'
RCUTILS_LOG_ERROR_NAMED(ROSBAG2_STORAGE_PACKAGE_NAME, __ss.str().c_str());
^~~~~~~~~~~~~~~~~~
/root/ros2_ws/ros2_base/install/include/rcutils/logging_macros.h:997:5: note: expanded from macro 'RCUTILS_LOG_ERROR_NAMED'
VA_ARGS)
^~~~~~~~~~~
/root/ros2_ws/ros2_base/install/include/rcutils/logging_macros.h:72:64: note: expanded from macro 'RCUTILS_LOG_COND_NAMED'
rcutils_log(&__rcutils_logging_location, severity, name, VA_ARGS);
^~~~~~~~~~~
/root/ros2_ws/ros2_ov/rosbag2/rosbag2_storage/src/rosbag2_storage/./impl/storage_factory_impl.hpp:99:7: note: treat the string as an argument to avoid this
/root/ros2_ws/ros2_ov/rosbag2/rosbag2_storage/include/rosbag2_storage/logging.hpp:40:59: note: expanded from macro 'ROSBAG2_STORAGE_LOG_ERROR_STREAM'
RCUTILS_LOG_ERROR_NAMED(ROSBAG2_STORAGE_PACKAGE_NAME, __ss.str().c_str());
^
In file included from /root/ros2_ws/ros2_ov/rosbag2/rosbag2_storage/src/rosbag2_storage/storage_factory.cpp:24:
/root/ros2_ws/ros2_ov/rosbag2/rosbag2_storage/src/rosbag2_storage/./impl/storage_factory_impl.hpp:112:7: error: format string is not a string literal (potentially insecure) [-Werror,-Wformat-security]
ROSBAG2_STORAGE_LOG_ERROR_STREAM(
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/root/ros2_ws/ros2_ov/rosbag2/rosbag2_storage/include/rosbag2_storage/logging.hpp:40:59: note: expanded from macro 'ROSBAG2_STORAGE_LOG_ERROR_STREAM'
RCUTILS_LOG_ERROR_NAMED(ROSBAG2_STORAGE_PACKAGE_NAME, __ss.str().c_str());
^~~~~~~~~~~~~~~~~~
/root/ros2_ws/ros2_base/install/include/rcutils/logging_macros.h:997:5: note: expanded from macro 'RCUTILS_LOG_ERROR_NAMED'
VA_ARGS)
^~~~~~~~~~~
/root/ros2_ws/ros2_base/install/include/rcutils/logging_macros.h:72:64: note: expanded from macro 'RCUTILS_LOG_COND_NAMED'
rcutils_log(&__rcutils_logging_location, severity, name, VA_ARGS);
^~~~~~~~~~~
/root/ros2_ws/ros2_ov/rosbag2/rosbag2_storage/src/rosbag2_storage/./impl/storage_factory_impl.hpp:112:7: note: treat the string as an argument to avoid this
/root/ros2_ws/ros2_ov/rosbag2/rosbag2_storage/include/rosbag2_storage/logging.hpp:40:59: note: expanded from macro 'ROSBAG2_STORAGE_LOG_ERROR_STREAM'
RCUTILS_LOG_ERROR_NAMED(ROSBAG2_STORAGE_PACKAGE_NAME, __ss.str().c_str());
^
In file included from /root/ros2_ws/ros2_ov/rosbag2/rosbag2_storage/src/rosbag2_storage/storage_factory.cpp:24:
/root/ros2_ws/ros2_ov/rosbag2/rosbag2_storage/src/rosbag2_storage/./impl/storage_factory_impl.hpp:131:7: error: format string is not a string literal (potentially insecure) [-Werror,-Wformat-security]
ROSBAG2_STORAGE_LOG_ERROR_STREAM(
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/root/ros2_ws/ros2_ov/rosbag2/rosbag2_storage/include/rosbag2_storage/logging.hpp:40:59: note: expanded from macro 'ROSBAG2_STORAGE_LOG_ERROR_STREAM'
RCUTILS_LOG_ERROR_NAMED(ROSBAG2_STORAGE_PACKAGE_NAME, __ss.str().c_str()); \

Subscriptions for record should be with info so that information like QoS can be obtained.

Description

Like in ROS 1, there should be a chance to control the replaying of a bag file.

spacebar should toggle the start/pause replaying.
s should solely spin the bag once.

Generally, as for now, there is no status feedback given when replaying a bag file. As a first step, there should be a message printed, whether the bag file is currently publishing data or not.

Completion criteria

• ros2 bag play uses keyboard_handler to receive key callbacks
• spacebar toggles pause/resume state for playback
• s does play_next
• ros2 bag play prints a help message when playback starts, explaining these controls

Related issues

#696

Implementation Notes

• May need to bloom-release keyboard_handler into Rolling - it is not yet in use.
• May need to expose keybindings to Python layer via Pybind - it makes sense for the CLI to define which keys do what, rather than hardcoding on the C++ end

Testing Notes

• May be difficult to do key-based end-to-end testing. It should be OK to mock the key callbacks

I wanted to test the performance of the current implementation, if its capable of writing multiple Raw cameradatastreams to my SSD.

Testusecase:

• 1xCamera
• Node to simulate datastream 1280x1040x3 Byte @ 30fps
  • rosidl error when i wanted to create a native image
  • Created data by creating a string with the corresponding size
• using sqlite plugin
• Ubuntu 18.04
• ROS2 packages / rosbag via apt installed

Result:

• 4,6 seconds recording: only 32 of ~138 images got saved

Did i miss something, or is rosbag2 not capable yet to save that amount of data?

Set the API for an abstract storage interface.

See the ros2/design#160 for reference.

The idea is to have a generic, common API to support multiple storage formats (sqlite, ros1bags, ...) to load binary data and the respective meta data to convert it later on to a functional ROS 2 message.

In order to easily release the current rosbag2 repos without a strong dependency on the ros1_bridge, this repo will be split in two.

• rosbag2 --> all default packages (sqlite3 plugin, Fast-RTPS dynamic CDR converter)
• rosbag2_v2 --> all ros1 rosbag v2 related packages

It shall be possible to convert binary data to ROS 2 messages from different RMW sources. If a bag file was recorded with one serialization format there shall be a convert function to deserialize this binary data to a ROS 2 message.

These various serialization formats shall be loaded via an plugin architecture.

currently the rosbag2_tests package depends on the rosbag2_bag_v2_plugin and thus can't be built without this. This also means that the ros1-agnostic rosbag tests can't be built.

The current play end-to-end tests cannot be executed reliably as the publishers are sending their messages before any subscriber can be matched in a deterministic fashion. In order to push the repos further towards the ros2.repos integration, we disabled this test. See conversation here:
#104 (comment)

This can be re-enabled when a more deterministic start of replaying a rosbag is available. One way to work towards this is to introduce a --paused option to start a rosbag, prepare all publishers but don't start publishing directly. This allows discovery to match with the respective subscribers to guarantee receiving all sent messages.

Not sure if rcutils_char_array_t is available in rosbag2_transport publish call

I'd like to add more plugins to the set of rosbag2_storage_default_plugins; and to improve code reuse, I'd like to do this by inheriting from existing plugins.

However, after creating a child class, I can't seem to configure pluginlib to utilize my appended plugin_description.xml, thus the get_interface_instance call fails to find the new plugin when I attempt to set the parent plugin's type as the child plugin's base_class_type instead of the parent's own base_class_type. Doing the latter of course prevents me from leaving the public function unimplemented when attempting to inherit.

$ ros2 bag play -s checkpoint bag_file_path
DISCLAIMER
ros2 bag is currently under development and not ready to use yet
[ERROR] [rosbag2_storage]: Could not load/open plugin with storage id 'checkpoint'.
[ERROR] [rosbag2_transport]: Failed to play: No storage could be initialized. Abort

Here is my current plugin_description.xml

<library path="rosbag2_storage_default_plugins">
  <class
    name="sqlite3"
    type="rosbag2_storage_plugins::SqliteStorage"
    base_class_type="rosbag2_storage::storage_interfaces::ReadWriteInterface"
  >
    <description>Plugin to write to SQLite3 databases</description>
  </class>
  <class
    name="checkpoint"
    type="rosbag2_storage_plugins::CheckpointStorage"
    base_class_type="rosbag2_storage_plugins::SqliteStorage"
  >
    <description>Plugin to write to Checkpoint databases</description>
  </class>
</library>

This commit/branch shows a minimal example of what was attempted above: 849c722

Currently, there is no way to terminate a play process other than waiting for its completion or a hard kill.
This is because signals are not properly processed and passed to the C++-layer.

Implement proper signal handling throughout rosbag2 (affecting at least ros2 bag play, possibly other commands).

The patches in this package suggest to me that it must always build the vendored project, in which case it must specify the dependencies of that project in order to build as a released package. The release job is currently failing due to the absence of pluginlib in the build container. A grep through the patches suggests there are likely other build and runtime dependencies that should be specified.

❯ git grep find_package
CMakeLists.txt:find_package(ament_cmake REQUIRED)
resources/cmakeliststxt.diff:+find_package(ament_cmake REQUIRED)
resources/cmakeliststxt.diff:+find_package(pluginlib 2.0 REQUIRED)
resources/cmakeliststxt.diff: find_package(console_bridge REQUIRED)
resources/cmakeliststxt.diff:-find_package(catkin REQUIRED COMPONENTS cpp_common pluginlib roscpp_serialization roscpp_traits rostime roslz4)
resources/cmakeliststxt.diff: find_package(Boost REQUIRED COMPONENTS date_time filesystem program_options regex)
resources/cmakeliststxt.diff: find_package(BZip2 REQUIRED)
resources/cmakeliststxt.diff:+find_package(ros1_bridge REQUIRED)
resources/cmakeliststxt.diff:+find_package(PkgConfig)
resources/cmakeliststxt.diff:-    find_package(rostest)
resources/cmakeliststxt.diff:+#    find_package(ament_cmake_gmock)
resources/rosbag1_encryption_patch.diff:     find_package(rostest)

Hi,
I am running Ubuntu 18.04 with the latest Melodic and Crystal installed from the repositories. I am trying to use the rosbag_v2 plugin to read in a ROS1 rosbag. In a terminal where I sourced the Crystal installation the plugin is not found:

user@user-VirtualBox:~/host$ ros2 bag info -s rosbag_v2 amsterdamer09.bag
DISCLAIMER
ros2 bag is currently under development and not ready to use yet
[ERROR] [rosbag2_storage]: Could not load/open plugin with storage id 'rosbag_v2'.
[ERROR] [rosbag2_transport]: Could not read metadata for amsterdamer09.bag. Please specify the path to the folder containing an existing 'metadata.yaml' file or provide correct storage id if metadata file doesn't exist (see help).

If I source both Crystal and Melodic (in that order) I get a segmentation fault when using ros2cli:

user@user-VirtualBox:~/host$ source /opt/ros/melodic/setup.bash
ROS_DISTRO was set to 'crystal' before. Please make sure that the environment does not mix paths from different distributions.

user@user-VirtualBox:~/host$ ros2 bag info -s rosbag_v2 amsterdamer09.bag
Failed to load entry point 'info': /opt/ros/crystal/lib/librosbag2_storage.so: undefined symbol: _ZNK12class_loader11ClassLoader15isLibraryLoadedEv
The C extension '/opt/ros/crystal/lib/python3.6/site-packages/rosbag2_transport/_rosbag2_transport_py.cpython-36m-x86_64-linux-gnu.so' failed to be imported while being present on the system. Please refer to 'https://github.com/ros2/ros2/wiki/Rclpy-Import-error-hint' for possible solutions
Segmentation fault (core dumped)

I already followed the proposed link to the Wiki and reiterated the installation instructions for Linux, but that didn't help. I would be thankful for some advice.

Options are:

• ros2 bag play <bag_file> --> play the complete bag file
• ros2 bag play <bag_file> -t topic1 [... topicN] --> play only specific topics from bag file

Connects to ros2/ros2#533

API shall be able to read and write ros 2 messages into a rosbag file.
It shall further be able to convert middleware dependent ros 2 messages into other encodings via available plugins.

In a second step these messages shall be recorded online and being published when the bag is getting replayed.

Todos:

• Basic interface for opening storage API and read/write data
• Plugin API for conversion API / loading message type_supports: see #11
• Transport layer for publishing data

I've been trying to build a rosbag2 storage plugin that is currently working with building against rosbag2 from source and have packaged separately from rosbag2_storage_default_plugins. However, when I try and build the same plugin package against the released binaries for crystal, I get some odd errors with colcon looking in the local workspace's install directory for package.sh files for packages that are already installed via the system. See the error except and reference dockerfile and plugin package below:

. /opt/ros/$ROS_DISTRO/setup.sh &&     apt-get update &&     rosdep install -q -y       --from-paths         src/rosbag2/rosbag2_storage_checkpoint_plugin       --ignore-src     && rm -rf /var/lib/apt/lists/*
...
executing command [apt-get install -y -qq ros-crystal-rosbag2-storage]
executing command [apt-get install -y -qq ros-crystal-rosbag2-storage-default-plugins]
executing command [apt-get install -y -qq ros-crystal-sqlite3-vendor]
executing command [apt-get install -y -qq ros-crystal-rosbag2-test-common]
#All required rosdeps installed successfully
Removing intermediate container a414f16f4c7b
 ---> 32bdfaafbdb0
Step 8/10 : ARG CMAKE_BUILD_TYPE=Release
 ---> Running in 3d7bcf664fc6
Removing intermediate container 3d7bcf664fc6
 ---> 110a131cfa19
Step 9/10 : RUN . /opt/ros/$ROS_DISTRO/setup.sh &&     colcon build       --symlink-install       --packages-select         rosbag2_storage_checkpoint_plugin       --cmake-args         -DCMAKE_BUILD_TYPE=$CMAKE_BUILD_TYPE
 ---> Running in 56a633e756e5
Starting >>> rosbag2_storage_checkpoint_plugin
[0.299s] ERROR:colcon.colcon_cmake.task.cmake.build:Failed to find the following files:
- /opt/ros_ws/install/rosbag2_test_common/share/rosbag2_test_common/package.sh
- /opt/ros_ws/install/sqlite3_vendor/share/sqlite3_vendor/package.sh
- /opt/ros_ws/install/rosbag2_storage/share/rosbag2_storage/package.sh
- /opt/ros_ws/install/rosbag2_storage_default_plugins/share/rosbag2_storage_default_plugins/package.sh
Check that the following packages have been built:
- rosbag2_test_common
- sqlite3_vendor
- rosbag2_storage
- rosbag2_storage_default_plugins
Failed   <<< rosbag2_storage_checkpoint_plugin	[ Exited with code 1 ]

Summary: 0 packages finished [0.17s]
  1 package failed: rosbag2_storage_checkpoint_plugin

FROM ros:crystal

# install ROS2 dependencies
RUN apt-get update && apt-get install -q -y \
      build-essential \
      cmake \
      git \
      python3-colcon-common-extensions \
      python3-vcstool \
      wget \
    && rm -rf /var/lib/apt/lists/*

# copy ros package repo
ENV ROS_WS /opt/ros_ws
RUN mkdir -p $ROS_WS/src
WORKDIR $ROS_WS
COPY ./ src/rosbag2/

# install dependency package dependencies
RUN . /opt/ros/$ROS_DISTRO/setup.sh && \
    apt-get update && \
    rosdep install -q -y \
      --from-paths \
        src/rosbag2/rosbag2_storage_checkpoint_plugin \
      --ignore-src \
    && rm -rf /var/lib/apt/lists/*

# build dependency package source
ARG CMAKE_BUILD_TYPE=Release
RUN . /opt/ros/$ROS_DISTRO/setup.sh && \
    colcon build \
      --symlink-install \
      --packages-select \
        rosbag2_storage_checkpoint_plugin \
      --cmake-args \
        -DCMAKE_BUILD_TYPE=$CMAKE_BUILD_TYPE

# source navigation2 workspace from entrypoint
RUN sed --in-place \
      's|^source .*|source "$ROS_WS/install/setup.bash"|' \
      /ros_entrypoint.sh

Connects to ros2/ros2#533

The storage API shall be ROS 2 agnostic and only handles the file access to read and write serialized binary code. It shall be able to describe the binary data with enough meta information to restore a complete message out of it.

Todos:

• implement abstract storage API interface: see #8
• factory loading the storage plugin to work with: see #7
• feature a SQLite3 plugin: see #9
• feature a ROS 1 bag plugin: see #10

The default storage plugin using an SQLite database currently writes every message to the database immediately. This has the benefit that on errors no messages are lost, but it incurs a performance overhead.

It would probably be best to use transactions and buffer messages, writing several at once. Since this behaviour is specific to the plugin, it might be best to implement a config for the storage plugin which can be edited by the user providing all parameters to tweak performance and memory usage.

Currently the rosbag2 layer provides only reader, writer, and info.
This interface shall be enhanced to also offer the possibility to read messages from a bag and potentially modify them and store the modified data in the original bag.

Hey,

i was finally able to successfully build rosbag2 on aarch64 with

colcon build --cmake-args -DBUILD_TESTING:BOOL=OFF

since LeakSanitizer is not available for arm64 Ubuntu 16.04 Xenial.

Afterwards i get the following error.

$ ros2 bag --help
Failed to load entry point 'play': cannot import name 'NODE_NAME_PREFIX'
Failed to load entry point 'record': cannot import name 'NODE_NAME_PREFIX'
usage: ros2 bag [-h] Call `ros2 bag <command> -h` for more detailed usage. ...

Various rosbag related sub-commands

optional arguments:
  -h, --help            show this help message and exit

Commands:
  info  ros2 bag info

  Call `ros2 bag <command> -h` for more detailed usage.

Has anyone ever stumbled across this problem and knows how to solve it?

Thanks in advance!

A plugin shall be available to load legacy ROS 1 bag files and to load binary and respective meta data.
The plugin has to adhere to the plugin API to be interchangeable with other plugins.

Required Info:

• Operating System:
  • Ubuntu 18.04
• Installation type:
  • source
• Version or commit hash:
  • Dashing
• DDS implementation:
  • Fast-RTPS

Expected behavior
I would be nice if messages published with the durability profile transient_local would be recorded with a subscriber that also uses the durability profile transient_local, so that messages published before the start of the recording get also recorded. This messages should then also be played back using the profile transient_local in order to keep up the latching behavior.

Currently, there is no way to know which converter and storage plugins are installed. This is annoying for storage plugins as rosbag_v2 as you need to know its name to play an old bagfile.

Provide a way to get a list of all installed plugins via command line.

I see a few options here:

• Try to provide autocompletion on -s (would be the nicest solution in my eyes)
• Add an option like --list-storage-plugins for ros2 bag play/record
• New verb for ros2 bag

Currently the ros2 bag info output is created directly in rosbag2_transport. This should change and the console output should happen in the python layer above.

Connects to ros2/ros2#533

Envisioned verb configurations

• add ros2 bag * as a new verb: see #7
• ros2 bag record [-a | topic1 [topic2 ... topicN]: see #13
• ros2 bag play <file_to_bag>: see #14
• ros2 bag info <file_to_bag>: see #15

Currently the rosbag2_record_all_integration_test test is flaky when running with FastRTPS and fails with the following error:

C++ exception with description "could not create publisher:
create_publisher() could not create publisher, at <ROS2_WS>/src/ros2/rmw_fastrtps/rmw_fastrtps_cpp/src/rmw_publisher.cpp:134, at <ROS2_WS>/src/ros2/rcl/rcl/src/rcl/publisher.c:173" 
thrown in the test body.
PARTICIPANT Error Type : test_msgs::msg::dds_::Primitives_ Not Registered -> Function createPublisher

@Karsten1987 Are we doing something wrong in rosbag2 or is this a rmw/ dds problem?

It should be possible to have data with different serialization formats (encodings) in the same bag.
All messages for a given topic, however, must be encoded with the same serialization format.

seen here and seems to be a consistent test timeout:
https://ci.ros2.org/job/ci_windows/5195/

Description

Feature request: Provide the --clock option so that rosbag2 publishes to the /clock topic alongside its messages.

Implementation Suggestion

Using starting_time of the bag and the stored (received) timestamp with every message, we can start a publisher for the /clock topic on playback and run it at a fixed rate, publishing messages when the next clock message is past their publishing time.

This requires no introspection or modification of the message contents themselves.

Note that player.cpp is already using message timestamps to sleep in between publishing, to approximate the original rates. In this loop (play_messages_until_queue_empty) we could potentially insert the clock publisher there.

Another option would be to put the clock publisher on a timer, and have it publish the last-published message time.

Original Ticket

From ros 1 I'm used to the --clock option in order that the topics have time stamps as If the system would run right now. I couldn't find this option in the documentation of rosbag2. How can I activate simulated time with rosbag2?

Currently, the rosbag_write_integration_test is flaky.

Valgrind and the debugger both suggest that cleanup of shared_ptr<SerializedBagMessage> does not work properly. However, the code never calls reset explicitly, nor is the memory in question released in any other way we can see.

Label-Question:
Do we have player and recorder functionality available to use like in rosbag?

Feature Request

• Detect QoS for topics in order to subscribe to all topics correctly
• Record QoS of topics in the bag
• Playback bags with recorded QoS
• Option to override the recorded QoS with a different profile on playback (e.g. recorded a BEST_EFFORT, but want to play back as RELIABLE)

Original issue report:

as the title says. When publishing topics with unreliable qos, the bag does not record any data on this topic as it expects a reliable connection - meaning all subscribers are spawned as reliable.

Work tracking

Tracking in-progress to-do list here

QoS storage in metadata

• QoS equality operators ros2/rclcpp#1032
• QoS YAML serialization #335
• QoS field add to metadata #335
• Query and store qos for a given topic at subscription time #333

Testing robustness

• Run tests on all rmw implementations
  • Fix small ament bug ament/ament_cmake#236
  • Run the rosbag2_transport tests on all rmw impls #349

Record with correct QoS

• Allow the GenericPublisher and GenericSubscription to receive and use a QoS profile #340
• Utilize queried qos to create subscription (if all offered profiles the same)
  • Initial implementation #343
  • disable tests because not working on cyclone #348
  • More holistic implementation #355

Playback with correct QoS

• Read saved QoS profiles from bag metadata into TopicMetadata #359
• Expose YAML metadata to the Player #377
• Provide adaptive QoS logic for publishing #379
• Use adaptive QoS to create publisher for playback #364

Explicit profile overrides

• Let user provide override QoS file to CLI for recording #341
• Let user provide override QoS file to CLI for playback #356
• Enable explicit per-topic qos profile override for recording #347
• Enable explicit per-topic qos profile override for playback #353
• ros2bag refactoring #358

The current design assumes to load a bag file with its top level folder and a hardcoded yaml file name.
This design shall be flexible though to load a bag file given a FQDN to a arbitrary yaml file without any hardcoded conventions.

Hi,

rcutils_get_error_string_safe is removed in ROS2 master workspace. It is still being used in rosbag2.

Below is the error I got while I tried record.

librosbag2_storage.so: undefined symbol: rcutils_get_error_string_safe

For rosbag2, we currently rely on subscribing serialized messages.
rmw_opensplice seems to not yet implement those functions in a working state.

The result is the same on both the Master and Crystal branch:

CMake Error at CMakeLists.txt:81 (find_package):
  By not providing "Findtest_msgs.cmake" in CMAKE_MODULE_PATH this project
  has asked CMake to find a package configuration file provided by
  "test_msgs", but CMake did not find one.

  Could not find a package configuration file provided by "test_msgs" with
  any of the following names:

    test_msgsConfig.cmake
    test_msgs-config.cmake

  Add the installation prefix of "test_msgs" to CMAKE_PREFIX_PATH or set
  "test_msgs_DIR" to a directory containing one of the above files.  If
  "test_msgs" provides a separate development package or SDK, be sure it has
  been installed.

Any hints on test_msgs?

See here:
https://github.com/ros2/rosbag2/blob/master/rosbag2/src/rosbag2/sequential_reader.cpp#L66

For every message in the bag file there is going to be a check whether or not this is has to be converted. I believe this check can be done once per topic and an appropriate reader shall be used. Whenever the message has to be converted, I believe it makes sense to have a separate class for it which always converts. On the other side, when no conversion is needed (which is most likely almost all time the case), the check should not be issued for every message.

When running the tests with valgrind, we can see that there is quite some memory leaked. It increases with every library loaded.

Some of the memory not being freed is due to the rcutils_loggig_macros.
I believe the rest of it is due to Pluginlib or Poco.

Options are:

• ros2 bag info <bag_file> --> shows info about the bag file

This info shall be similar to the existing ROS 1 info:

$ rosbag info foo.bag
path:        foo.bag
version:     2.0
duration:    1.2s
start:       Jun 17 2010 14:24:58.83 (1276809898.83)
end:         Jun 17 2010 14:25:00.01 (1276809900.01)
size:        14.2 KB
messages:    119
compression: none [1/1 chunks]
types:       geometry_msgs/Point [4a842b65f413084dc2b10fb484ea7f17]
topics:      /points   119 msgs @ 100.0 Hz : geometry_msgs/Point

I've been looking into potentially using rosbag2 to convert something like the KITTI dataset into a rosbag2 format. There are two ways to go about it, one is to convert and publish the data then use ros2bag record to record it, the other is to simply do the conversion and write directly to the bag using the rosbag2 API.

I might be mistaken but it seems that currently rosbag2 does not expose an intuitive interface for easy writing of rosbags. Looking at the writer itself for the record functionality, the API directly writes the serialized message into the bag. There should be, similar to the rosbag1 API, a way to simply pass in the deserialized msg directly and have the API handle the serialization and the writing.

Decide which parameters should be accessible via the CLI and provide access.
Candidates:

• qos settings for publishers and subscribers
• read_ahead_queue_size and lower_bound for the message queue during play
• read_ahead_sleep_time for filling the message queue during play