eprosima / dds-record-replay Goto Github PK

eProsima DDS Record & Replay is an end-user software application that efficiently saves DDS data published into a DDS environment in a MCAP format database. Thus, the exact playback of the recorded network events is possible. Looking for commercial support? Contact [email protected]

Home Page: https://eprosima.com

License: Apache License 2.0

CMake 3.76% C++ 92.99% SWIG 1.17% Python 1.86% Batchfile 0.03% C 0.19%

dds eprosima fastdds recorder replay

dds-record-replay's People

Contributors

Stargazers

Watchers

Forkers

rsanchez15 ihilt contr4l figurerobotics erich71 guoyuqiang123 fujitatomoya ryan-red

dds-record-replay's Issues

main v0.3.0 does not build

hello,

main does not build because it tries to build fastcdr / fastrtps v1 and v2.

I think this is because:
In cmake_utils/cmake/compilation/compile_library.cmake

        if ("${ARGV2}" STREQUAL "")
            file(
                GLOB_RECURSE SOURCES_FILES
                    "${_SOURCE_PATH}/*.c"
                    "${_SOURCE_PATH}/*.cpp"
                    "${_SOURCE_PATH}/*.cxx"
                    "${_SOURCE_PATH}/**/*.c"
                    "${_SOURCE_PATH}/**/*.cpp"
                    "${_SOURCE_PATH}/**/*.cxx"

so when calling:

# Compile C++ library
compile_library(
    "${PROJECT_SOURCE_DIR}/src/cpp" # Source directory
    "${PROJECT_SOURCE_DIR}/include" # Include directory
    "${SOURCES_FILES}"
    "${HEADERS_FILES}"
)

it will also get the files under V1 and V2 at compile time.

Shapedemo works but customized topic

LaserScan.idl from ROS2.

-typeobject ：1. can be recorded but foxglove can‘t online rendering under “3D” panel. 2. The mcap file recored can be offline replay by foxglove but can’t rendering with “3D” panel. 3.pubsubtypename should be changed from ”namespace::name” to “namespace::dds_::name_”

-typeros2 ：1. can be online rendering with foxglove “3D” panel but can't be recorded which prompting “not yet available” .

Need help on this, thanks

Interoperability with other DDS implementations

I gave the Recorder a try with RTI's shapes demo and found that while something seemed to end up recorded in the mcap, it was unloadable within Foxglove, which gave an error that the Shapes type schema was missing when I imported the file. I tried various permutations of these settings within my config file, but it didn't change anything:

record-types: true
ros2-types: true

I don't know if this is an issue between eProsima's Recorder and Foxglove, or between RTI Connext (or one of its protocol-related QOS policies) and eProsima DDS and/or Recorder.

I also didn't necessarily expect Recorder to be compatible with other vendors, but I was hoping it would work with Connext and/or Cyclone, as we work with orgs/software that could use any of the three major implementations.

If this isn't a Foxglove specific thing, could someone share the DDS standard specifications/features/QOS policy configurations that Recorder requires to properly function?

Thanks!

requirement

1 ddsrecorder should be customize the file storage path individually.
2 ddsrecorder_controller should be possible to customize the filename remotely.
3 ddsrecorder should be possible to set the initialization state,for example, the initial state is stop.
Thank you!

DDS-Record-Replay supports `info` and `list` command to peek the mcap file?

Description / Questions

provides many configuration for record and replay, most likely related to DDS configuration.

on the other hand, doc does not say anything about the command line interface to check the recorded data file to peek, e.g ros2 bag info or ros2 bag list.
those features are really user-friendly after recording and before playing to see what is stored in the recorded data.

if mcap CLI is supposed to be used in these use cases since it uses mcap underneath data format, i believe that it should be explained in user manual as well, including mcap CLI installation. (https://mcap.dev/guides/cli)

core dump

I am encountering an issue with a core dump when my IDL file includes an enumeration type definition, and I have created a corresponding DataWriter in my C++ program. When running this program and attempting to run DDS Recorder, I intermittently experience a core dump.
I am certain that the enumeration type is causing the issue. If I do not use an enumeration type, a core dump does not occur.

test step

run c++ program which create a datawrite,and keep it run.
run the shell script

 #!/bin/bash
origin_path=`pwd`
count=1
source /opt/FastDDS/DDS-Record-Replay/local_setup.sh
until [ $count -gt 100 ]; do
    sleep 1
    nohup /opt/FastDDS/DDS-Record-Replay/ddsrecorder_tool/bin/ddsrecorder -d -c ../data/config/recorder_config.yaml>/dev/null 2>&1 &
    sleep 1
    pkill -SIGINT -u $(whoami) ddsrecorder
    echo "Iteration $count"
    ((count++))
done
cd ${origin_path}

.mcap cannot be displayed in foxglove studio

Is there an already existing issue for this?

I have searched the existing issues

Expected behavior

.mcap is displayed normally in foxglove studio

Current behavior

When I use foxglove studio to play back mcap files, some can be displayed normally, while others cannot.

Steps to reproduce

1.Execute the fastddsgen command to generate idl code.(./fastddsgen *.idl -typeobject)
2.Copy the generated code to the shapes-demo code project and compile it
3../ShapeDemo

4../ddsrecorder

5.Use foxglove studio to play back .mcap files

issues.zip

Fast DDS version/commit

fastdds:2.11.0
foxglove studio:1.64.0

Platform/Architecture

Ubuntu Focal 20.04 amd64

Transport layer

Default configuration, UDPv4 & SHM

Additional context

No response

XML configuration file

No response

Relevant log output

No response

Network traffic capture

No response

DDSRecord fails to copy Hello World data from TypeLookupService

OS used:

Ubuntu 18.04

Arch:

aarch64

Build steps:

#!/bin/bash

set -e

ROOT_DIR=/media/nvme/ddsrecord-replay

git clone https://github.com/lz4/lz4.git
cd lz4
git checkout v1.9.3
make -j8
make install
cd -

git clone https://github.com/facebook/zstd.git
cd zstd
git checkout v1.4.8
make -j8
make install
cd -

rm -rf ${ROOT_DIR}/DDS-Record-Replay > /dev/null

mkdir -p ${ROOT_DIR}/DDS-Record-Replay/src
mkdir -p ${ROOT_DIR}/DDS-Record-Replay/build
cd ${ROOT_DIR}/DDS-Record-Replay
wget https://raw.githubusercontent.com/eProsima/DDS-Record-Replay/main/ddsrecordreplay.repos
vcs import src < ddsrecordreplay.repos

cd ${ROOT_DIR}/DDS-Record-Replay
mkdir build/foonathan_memory_vendor
cd build/foonathan_memory_vendor
cmake ${ROOT_DIR}/DDS-Record-Replay/src/foonathan_memory_vendor -DCMAKE_INSTALL_PREFIX=${ROOT_DIR}/DDS-Record-Replay/install -DBUILD_SHARED_LIBS=ON
cmake --build . --target install -- -j8

cd ${ROOT_DIR}/DDS-Record-Replay
mkdir build/fastcdr
cd build/fastcdr
cmake ${ROOT_DIR}/DDS-Record-Replay/src/fastcdr -DCMAKE_INSTALL_PREFIX=${ROOT_DIR}/DDS-Record-Replay/install
cmake --build . --target install -- -j8

cd ${ROOT_DIR}/DDS-Record-Replay
mkdir build/fastdds
cd build/fastdds
cmake ${ROOT_DIR}/DDS-Record-Replay/src/fastdds -DCMAKE_INSTALL_PREFIX=${ROOT_DIR}/DDS-Record-Replay/install -DCMAKE_PREFIX_PATH=${ROOT_DIR}/DDS-Record-Replay/install
cmake --build . --target install -- -j8

# CMake Utils
cd ${ROOT_DIR}/DDS-Record-Replay
mkdir build/cmake_utils
cd build/cmake_utils
cmake ${ROOT_DIR}/DDS-Record-Replay/src/dev-utils/cmake_utils -DCMAKE_INSTALL_PREFIX=${ROOT_DIR}/DDS-Record-Replay/install -DCMAKE_PREFIX_PATH=${ROOT_DIR}/DDS-Record-Replay/install
cmake --build . --target install -- -j8

# C++ Utils
cd ${ROOT_DIR}/DDS-Record-Replay
mkdir build/cpp_utils
cd build/cpp_utils
cmake ${ROOT_DIR}/DDS-Record-Replay/src/dev-utils/cpp_utils -DCMAKE_INSTALL_PREFIX=${ROOT_DIR}/DDS-Record-Replay/install -DCMAKE_PREFIX_PATH=${ROOT_DIR}/DDS-Record-Replay/install
cmake --build . --target install -- -j8

# ddspipe_core
cd ${ROOT_DIR}/DDS-Record-Replay
mkdir build/ddspipe_core
cd build/ddspipe_core
cmake ${ROOT_DIR}/DDS-Record-Replay/src/ddspipe/ddspipe_core -DCMAKE_INSTALL_PREFIX=${ROOT_DIR}/DDS-Record-Replay/install -DCMAKE_PREFIX_PATH=${ROOT_DIR}/DDS-Record-Replay/install
cmake --build . --target install -- -j8

# ddspipe_participants
cd ${ROOT_DIR}/DDS-Record-Replay
mkdir build/ddspipe_participants
cd build/ddspipe_participants
cmake ${ROOT_DIR}/DDS-Record-Replay/src/ddspipe/ddspipe_participants -DCMAKE_INSTALL_PREFIX=${ROOT_DIR}/DDS-Record-Replay/install -DCMAKE_PREFIX_PATH=${ROOT_DIR}/DDS-Record-Replay/install
cmake --build . --target install -- -j8

# ddspipe_yaml
cd ${ROOT_DIR}/DDS-Record-Replay
mkdir build/ddspipe_yaml
cd build/ddspipe_yaml
cmake ${ROOT_DIR}/DDS-Record-Replay/src/ddspipe/ddspipe_yaml -DCMAKE_INSTALL_PREFIX=${ROOT_DIR}/DDS-Record-Replay/install -DCMAKE_PREFIX_PATH=${ROOT_DIR}/DDS-Record-Replay/install
cmake --build . --target install -- -j8

# ddsrecorder_participants
cd ${ROOT_DIR}/DDS-Record-Replay
mkdir build/ddsrecorder_participants
cd build/ddsrecorder_participants
cmake ${ROOT_DIR}/DDS-Record-Replay/src/ddsrecordreplay/ddsrecorder_participants -DCMAKE_INSTALL_PREFIX=${ROOT_DIR}/DDS-Record-Replay/install -DCMAKE_PREFIX_PATH=${ROOT_DIR}/DDS-Record-Replay/install
cmake --build . --target install -- -j8

# ddsrecorder_yaml
cd ${ROOT_DIR}/DDS-Record-Replay
mkdir build/ddsrecorder_yaml
cd build/ddsrecorder_yaml
cmake ${ROOT_DIR}/DDS-Record-Replay/src/ddsrecordreplay/ddsrecorder_yaml -DCMAKE_INSTALL_PREFIX=${ROOT_DIR}/DDS-Record-Replay/install -DCMAKE_PREFIX_PATH=${ROOT_DIR}/DDS-Record-Replay/install
cmake --build . --target install -- -j8

# ddsrecorder
cd ${ROOT_DIR}/DDS-Record-Replay
mkdir build/ddsrecorder_tool
cd build/ddsrecorder_tool
cmake ${ROOT_DIR}/DDS-Record-Replay/src/ddsrecordreplay/ddsrecorder -DCMAKE_INSTALL_PREFIX=${ROOT_DIR}/DDS-Record-Replay/install -DCMAKE_PREFIX_PATH=${ROOT_DIR}/DDS-Record-Replay/install
cmake --build . --target install -- -j8

# ddsreplayer
cd ${ROOT_DIR}/DDS-Record-Replay
mkdir build/ddsreplayer_tool
cd build/ddsreplayer_tool
cmake ${ROOT_DIR}/DDS-Record-Replay/src/ddsrecordreplay/ddsreplayer -DCMAKE_INSTALL_PREFIX=${ROOT_DIR}/DDS-Record-Replay/install -DCMAKE_PREFIX_PATH=${ROOT_DIR}/DDS-Record-Replay/install
cmake --build . --target install -- -j8

TypeLookupService output:

./TypeLookupService
Participant < 01.0f.b6.0f.7c.04.54.88.00.00.00.00|0.0.1.c1> created...
        - DDS Domain: 0
        - DataWriter: 01.0f.b6.0f.7c.04.54.88.00.00.00.00|0.0.1.3
        - Topic name: /dds/topic
        - Topic data type: HelloWorld
Publisher running. Please press CTRL+C to stop the Publisher at any time.
Message published: 
index: 0
message: Hello World
-----------------------------------------------------
Message published: 
index: 1
message: Hello World
-----------------------------------------------------
Message published: 
index: 2
message: Hello World
-----------------------------------------------------
Message published: 
index: 3
message: Hello World
-----------------------------------------------------
Message published: 
index: 4
message: Hello World
-----------------------------------------------------
Message published: 
index: 5
message: Hello World
-----------------------------------------------------
Message published: 
index: 6
message: Hello World
-----------------------------------------------------
Message published: 
index: 7
message: Hello World
-----------------------------------------------------
Message published: 
index: 8
message: Hello World
-----------------------------------------------------
Message published: 
index: 9
message: Hello World
-----------------------------------------------------
Message published: 
index: 10
message: Hello World
-----------------------------------------------------
Message published: 
index: 11
message: Hello World
-----------------------------------------------------
Message published: 
index: 12
message: Hello World
-----------------------------------------------------
Message published: 
index: 13
message: Hello World
-----------------------------------------------------
Message published: 
index: 14
message: Hello World
-----------------------------------------------------
2023-05-25 17:01:12.123 [RTPS_TRANSPORT_SHM Warning] Buffer is being invalidated, segment_size may be insufficient -> Function invalidate_if_not_processing
DataWriter matched with DataReader: 1.f.b6.f.e7.4.f.8b.0.0.0.0.0.0.1.4
Message published: 
index: 15
message: Hello World
-----------------------------------------------------
Message published: 
index: 16
message: Hello World
-----------------------------------------------------

ddsrecord output:

./ddsrecorder 
Not configuration file given, using default file DDS_RECORDER_CONFIGURATION.yaml.
Starting DDS Recorder execution.
DDS Recorder running.
^C./ddsrecorder: symbol lookup error: /media/nvme/ddsrecord-replay/DDS-Record-Replay/install/lib/libddsrecorder_participants.so.0: undefined symbol: ZSTD_compress2

File created by ddsrecord:

ls -la output_
2023-05-25_21-01-12.mcap.tmp~ 
-rw-r--r-- 1 root root 0 May 25 17:01 output_2023-05-25_21-01-12.mcap.tmp~

Recorder Config used:

dds:
  domain: 0

recorder:
  output:
    filename: "output"
    path: "."

  buffer-size: 10
  event-window: 60
  log-publish-time: false
  downsampling: 1
  max-reception-rate: 2000

remote-controller:
  enable: false
  domain: 0
  initial-state: "RUNNING"
  command-topic-name: "/ddsrecorder/command"
  status-topic-name: "/ddsrecorder/status"

specs:
  threads: 8
  max-pending-samples: 10
  cleanup-period: 90

Please note that sometimes ddsrecord create non-zero size files and does not complain about the "undefined symbol: ZSTD_compress2", but when trying to replay the subscriber does not detect any messages.

versions of liblz4 and libzstd to use

I am trying to build according to your instructions at https://dds-recorder.readthedocs.io/en/latest/rst/developer_manual/installation/sources/linux.html but I keep getting compile errors when building mcap due to LZ4 and zstd libraries and I suspect it is a version incompatibility.

I am using ubuntu 18.04 and I installed libzstd-dev and liblz4-dev using the package manager which did not work so I build the libraries from source and now I am getting different compile errors. Knowing which version I should build would help me resolve this issue, thanks.

win11 can not replay

Environment: Windows 11
Issue:

C:\record-replayer-install\bin\ddsreplayer.exe --log-verbosity "warning" -i C:\Users\xxxxx\Downloads\output_2023-07-24_08-38-53.mcap return directly without pub any message.
"[RTPS_TRANSPORT_SHM Error] Failed init_port fastrtps_port7415 " sometime will prompt this message.

Version:
C:\record-replayer-install\bin\ddsreplayer.exe -v
DDS Record & Replay v0.2.0
commit hash: 66492d1

Compile ENV:
C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Visual Studio 2019\Visual Studio Tools\VC\x64 Native Tools Command Prompt for VS 2019

Cmake:
C:\Users\xxxxxxxx>cmake --version
cmake version 3.24.1
CMake suite maintained and supported by Kitware (kitware.com/cmake).

wget https://raw.githubusercontent.com/eProsima/DDS-Record-Replay/main/ddsrecordreplay.repos ddsrecordreplay.repos

controller & ddsrecorder.exe seems good but ddsreplayer.exe which need help.

Main does not build with latest FastDDS & FastCDR

Unable to build main with latest FastDDS (Fast DDS v2.12.0) and latest Fast-CDR (Fast CDR v2.0.0) due to major changes in CDR.

Any plans for an upcoming release to support Fast CDR v2.0.0 ?

`colcon build` failure for fastdds

Description

Walking through https://dds-recorder.readthedocs.io/en/latest/rst/developer_manual/installation/sources/linux.html#colcon-installation-recommended, eventually failed to compile fastdds with incompatibility with fastcdr as below.

Build Failure

colcon build --symlink-install
...<snip>
Starting >>> fastrtps
--- stderr: py_utils                                                           
/usr/lib/python3/dist-packages/setuptools/command/easy_install.py:158: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
/usr/lib/python3/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
/usr/lib/python3/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
---
Finished <<< py_utils [0.66s]
--- stderr: fastrtps                             
In file included from /root/docker_ws/dds_ws/DDS-Record-Replay/src/fastdds/include/fastdds/dds/topic/IContentFilter.hpp:30,
                 from /root/docker_ws/dds_ws/DDS-Record-Replay/src/fastdds/src/cpp/fastdds/topic/DDSSQLFilter/DDSFilterExpression.hpp:27,
                 from /root/docker_ws/dds_ws/DDS-Record-Replay/src/fastdds/src/cpp/fastdds/topic/DDSSQLFilter/DDSFilterExpression.cpp:19:
/root/docker_ws/dds_ws/DDS-Record-Replay/src/fastdds/include/fastrtps/types/TypesBase.h:273:27: error: expected unqualified-id before numeric constant
  273 | #define MEMBER_ID_INVALID 0X0FFFFFFF
      |                           ^~~~~~~~~~
In file included from /root/docker_ws/dds_ws/DDS-Record-Replay/src/fastdds/src/cpp/fastdds/topic/DDSSQLFilter/DDSFilterExpression.cpp:30:
/root/docker_ws/dds_ws/DDS-Record-Replay/install/fastcdr/include/fastcdr/Cdr.h: In member function ‘eprosima::fastcdr::Cdr& eprosima::fastcdr::Cdr::operator<<(const _T&)’:
/root/docker_ws/dds_ws/DDS-Record-Replay/install/fastcdr/include/fastcdr/Cdr.h:328:31: error: no match for ‘operator==’ (operand types are ‘int’ and ‘eprosima::fastcdr::MemberId’)
  328 |         if (MEMBER_ID_INVALID == next_member_id_)
      |                               ^~ ~~~~~~~~~~~~~~~
      |                                  |
      |                                  eprosima::fastcdr::MemberId
/root/docker_ws/dds_ws/DDS-Record-Replay/install/fastcdr/include/fastcdr/Cdr.h: In member function ‘eprosima::fastcdr::Cdr& eprosima::fastcdr::Cdr::operator>>(_T&)’:
/root/docker_ws/dds_ws/DDS-Record-Replay/install/fastcdr/include/fastcdr/Cdr.h:358:31: error: no match for ‘operator==’ (operand types are ‘int’ and ‘eprosima::fastcdr::MemberId’)
  358 |         if (MEMBER_ID_INVALID == next_member_id_)
      |                               ^~ ~~~~~~~~~~~~~~~
      |                                  |
      |                                  eprosima::fastcdr::MemberId
/root/docker_ws/dds_ws/DDS-Record-Replay/src/fastdds/src/cpp/fastdds/topic/DDSSQLFilter/DDSFilterExpression.cpp: In member function ‘virtual bool eprosima::fastdds::dds::DDSSQLFilter::DDSFilterExpression::evaluate(const SerializedPayload&, const eprosima::fastdds::dds::IContentFilter::FilterSampleInfo&, const GUID_t&) const’:
/root/docker_ws/dds_ws/DDS-Record-Replay/src/fastdds/src/cpp/fastdds/topic/DDSSQLFilter/DDSFilterExpression.cpp:58:95: error: ‘DDS_CDR’ is not a member of ‘eprosima::fastcdr::Cdr’
   58 |         Cdr deser(fastbuffer, eprosima::fastcdr::Cdr::DEFAULT_ENDIAN, eprosima::fastcdr::Cdr::DDS_CDR);
      |                                                                                               ^~~~~~~
gmake[2]: *** [src/cpp/CMakeFiles/fastrtps.dir/build.make:90: src/cpp/CMakeFiles/fastrtps.dir/fastdds/topic/DDSSQLFilter/DDSFilterExpression.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:941: src/cpp/CMakeFiles/fastrtps.dir/all] Error 2
gmake: *** [Makefile:146: all] Error 2
---
Failed   <<< fastrtps [4.01s, exited with code 2]
                                 
Summary: 6 packages finished [4.53s]
  1 package failed: fastrtps
  2 packages had stderr output: fastrtps py_utils
  11 packages not processed

Additional Information

repo file

root@tomoyafujita:~/docker_ws/dds_ws/DDS-Record-Replay# cat ddsrecordreplay.repos 
repositories:
    foonathan_memory_vendor:
        type: git
        url: https://github.com/eProsima/foonathan_memory_vendor.git
        version: master
    fastcdr:
        type: git
        url: https://github.com/eProsima/Fast-CDR.git
        version: master
    fastdds:
        type: git
        url: https://github.com/eProsima/Fast-DDS.git
        version: bugfix/complex-dynamic-types
    dev-utils:
        type: git
        url: https://github.com/eProsima/dev-utils.git
        version: main
    ddspipe:
        type: git
        url: https://github.com/eProsima/DDS-Pipe.git
        version: main
    ddsrecordreplay:
        type: git
        url: https://github.com/eProsima/DDS-Record-Replay.git
        version: main

eProsima/Fast-DDS@master...bugfix/complex-dynamic-types should be used?

as far as i checked with https://fast-dds.docs.eprosima.com/en/latest/installation/sources/sources_linux.html and 824e664, master branches should have no problem between them.

eprosima / dds-record-replay Goto Github PK

dds-record-replay's People

Contributors

Stargazers

Watchers

Forkers

dds-record-replay's Issues

Description / Questions

Is there an already existing issue for this?

Expected behavior

Current behavior

Steps to reproduce

Fast DDS version/commit

Platform/Architecture

Transport layer

Additional context

XML configuration file

Relevant log output

Network traffic capture

OS used:

Arch:

Build steps:

TypeLookupService output:

ddsrecord output:

File created by ddsrecord:

Recorder Config used:

Description

Build Failure

Additional Information

Recommend Projects

Recommend Topics

Recommend Org