robotperf / benchmarks Goto Github PK

View Code? Open in Web Editor NEW

144.0 144.0 16.0 21.22 MB

Benchmarking suite to evaluate 🤖 robotics computing performance. Vendor-neutral. ⚪Grey-box and ⚫Black-box approaches.

Home Page: https://robotperf.net

License: Apache License 2.0

Dockerfile 0.10% CMake 2.52% C 1.45% C++ 1.62% Python 94.25% Shell 0.05%

acceleration benchmarking cpu fpga gpu performance robotics ros2 tpu

benchmarks's People

Contributors

Stargazers

Watchers

Forkers

araujorayza jasonjabbour vmayoral russ76 mpstewart1 mich3lls umbaji001 mcres alexmarfar prateek-ari leo47007 siliciuss arron-indrotek abirhossen786 juanma-rm dtbinh

benchmarks's Issues

Add a5_resize benchmark

The a5_resize benchmark will take an image as input and perform a resize operation on it.

[NEW BENCHMARK] depthimage_to_laserscan

Specify the ROS2 package are you benchmarking:
depthimage_to_laserscan

Provide the link to the ROS2 package:
https://github.com/ros-perception/depthimage_to_laserscan/tree/ros2

Which category does your benchmark fall under?

What metric are you benchmarking?
latency, power, throughput

What is your estimated date of completion?
July 18th, 2023

What hardware will your reference implementation run on?
12th Gen Intel(R) Core(TM) i9-12900KF

Your organization:
Harvard University

Your name:
Jason Jabbour

Your email address or contact information:
[email protected]

[NEW BENCHMARK] a6 resize throughput

Specify the ROS2 package are you benchmarking:
image_pipeline

Provide the link to the ROS2 package:
https://github.com/ros-perception/image_pipeline/tree/humble/

Which category does your benchmark fall under?

What metric are you benchmarking?
throughput (msgs/s and/or bytes/s) as defined in REP-2014

What is your estimated date of completion?
May 22nd, 2023

What hardware will your reference implementation run on?
- 12th Gen Intel(R) Core(TM) i7-12700H x 20
- NVIDIA GeForce RTX 3060 Max-Q

Your organization:
Acceleration Robotics

Your name:
Alejandra M. Fariña

Your email address or contact information:
[email protected]

[NEW BENCHMARK] d3_collision_checking_bullet

Specify the ROS2 package are you benchmarking:
moveit2

Provide the link to the ROS2 package:
https://github.com/ros-planning/moveit2

Which category does your benchmark fall under?

What metric are you benchmarking?
Latency and power

What is your estimated date of completion?
July 21th, 2023

What hardware will your reference implementation run on?
Workstation

Your organization:
Acceleration Robotics

Your name:
Martiño Crespo Álvarez

Your email address or contact information:
[email protected]

power_msgs package for ROS2 Humble

Hi! I am trying to use the robotperf ros2_benchmark repo to create the auto.launch.py script for the a6 benchmark. It seems like the ros2_benchmark package depends on the power_msgs package.

However, I can't seem to find the Humble version of power_msgs.

Any tips about how I can install this package?

Thanks!!

[NEW BENCHMARK] d5_inverse_kinematics_lma

Specify the ROS2 package are you benchmarking:
moveit2

Provide the link to the ROS2 package:
https://github.com/ros-planning/moveit2

Which category does your benchmark fall under?

What metric are you benchmarking?
Latency and power

What is your estimated date of completion?
July 27th, 2023

What hardware will your reference implementation run on?
Workstation

Your organization:
Acceleration Robotics

Your name:
Martiño Crespo Álvarez

Your email address or contact information:
[email protected]

[NEW BENCHMARK] d4_inverse_kinematics_kdl

Specify the ROS2 package are you benchmarking:
moveit2

Provide the link to the ROS2 package:
https://github.com/ros-planning/moveit2

Which category does your benchmark fall under?

What metric are you benchmarking?
Latency and power

What is your estimated date of completion?
July 27th, 2023

What hardware will your reference implementation run on?
Workstation

Your organization:
Acceleration Robotics

Your name:
Martiño Crespo Álvarez

Your email address or contact information:
[email protected]

[NEW BENCHMARK] c2_diffbot_diff_driver_controller

Specify the ROS2 package are you benchmarking:
diff_drive_controller

Provide the link to the ROS2 package:
https://github.com/ros-controls/ros2_controllers/tree/humble/diff_drive_controller

Which category does your benchmark fall under?

What metric are you benchmarking?
Latency and power

What is your estimated date of completion?
June 30th, 2023

What hardware will your reference implementation run on?
Workstation

Your organization:
Acceleration Robotics

Your name:
Martiño Crespo Álvarez

Your email address or contact information:
[email protected]

[NEW BENCHMARK] d1_xarm6_planning_and_traj_execution

Specify the ROS2 package are you benchmarking:
moveit2

Provide the link to the ROS2 package:
https://github.com/ros-planning/moveit2

Which category does your benchmark fall under?

What metric are you benchmarking?
Latency and power

What is your estimated date of completion?
July 14th, 2023

What hardware will your reference implementation run on?
Workstation

Your organization:
Acceleration Robotics

Your name:
Martiño Crespo Álvarez

Your email address or contact information:
[email protected]

Refactoring of benchmarks analysis scripts

Currently, the analysis files for a1, a2, a3 and a5 is duplicated.

The objective is to create a benchmark_utilities package that is imported and used by every analysis script, avoiding the current situation.

a1 benchmark, preliminary results on Intel x86_64

This ticket presents preliminary results of RobotPerf benchmark a1 in a development workstation Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz, 64 GB RAM. The purpose of these results is to provide a developer preview and align the building blocks of RobotPerf, including benchmarks and rosbags. In the near future results like this will be automatically pushed into the corresponding's benchmark.yaml file, and a series of CI/CD jobs will update the READMEs*

ID	Graph	Benchmark summary	Datasource	Metric	🖥️ Intel x86_64¹
`a1`		Perception computational graph composed by 2 dataflow-connected Components, `rectify` and `resize`.	perception/image	latency (ms)	21.08 ms _{^(31-01-2023)}

Additional information

Barcharts	Trace

	Benchmark Mean	Benchmark RMS	Benchmark Max	Benchmark Min	Mean	RMS	Max	Min
RobotPerf benchmark	13.67 ms	13.92 ms	21.08 ms	6.83 ms	13.92 ms	14.15 ms	21.40 ms	6.92 ms

Note: significant errors encountered while working with FastDDS. Specially on embedded. Moved to CycloneDDS for benchmarking purposes.

Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz, 64 GB RAM ↩

[NEW BENCHMARK] b3_apriltag_detection

Specify the ROS2 package are you benchmarking:
isaac_ros_apriltag

Provide the link to the ROS2 package:
https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_apriltag

Which category does your benchmark fall under?

What metric are you benchmarking?
Latency, throughput and power

June 23rd

What hardware will your reference implementation run on?
Workstation and AGX Orin

Your organization:
Acceleration Robotics

Your name:
Martiño Crespo Álvarez

Your email address or contact information:
[email protected]

[NEW BENCHMARK] c3_rrbot_forward_command_controller

Specify the ROS2 package are you benchmarking:
forward_command_controller

Provide the link to the ROS2 package:
https://github.com/ros-controls/ros2_controllers/tree/humble/forward_command_controller

Which category does your benchmark fall under?

What metric are you benchmarking?
Latency and power

What is your estimated date of completion?
June 30th, 2023

What hardware will your reference implementation run on?
Workstation

Your organization:
Acceleration Robotics

Your name:
Martiño Crespo Álvarez

Your email address or contact information:
[email protected]

[NEW BENCHMARK] c1_rrbot_joint_trajectory_controller

Specify the ROS2 package are you benchmarking:
joint_trajectory_controller

Provide the link to the ROS2 package:
https://github.com/ros-controls/ros2_controllers/tree/humble/joint_trajectory_controller

Which category does your benchmark fall under?

What metric are you benchmarking?
Latency and power

What is your estimated date of completion?
June 30th, 2023

What hardware will your reference implementation run on?
Workstation

Your organization:
Acceleration Robotics

Your name:
Martiño Crespo Álvarez

Your email address or contact information:
[email protected]

Any pypi package?

[NEW BENCHMARK] imu_transformer

Specify the ROS2 package are you benchmarking:
imu_transformer

Provide the link to the ROS2 package:
https://github.com/ros-perception/imu_pipeline/tree/ros2/imu_transformer

Which category does your benchmark fall under?

What metric are you benchmarking?
latency, power, throughput

What is your estimated date of completion?
July 18th, 2023

What hardware will your reference implementation run on?
12th Gen Intel(R) Core(TM) i9-12900KF

Your organization:
Harvard University

Your name:
Jason Jabbour

Your email address or contact information:
[email protected]

[NEW BENCHMARK] b2_map_localization

Specify the ROS2 package are you benchmarking:
map_localization

Provide the link to the ROS2 package:
https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_map_localization

Which category does your benchmark fall under?

What metric are you benchmarking?
Latency, throughput and power

What is your estimated date of completion?
June 23rd

What hardware will your reference implementation run on?
Workstation

Your organization:
Acceleration Robotics

Your name:
Alejandra Martínez Fariña

Your email address or contact information:
[email protected]

a1_perception_node not reproducable

Hey, so I had the following doubts after going through the benchmark repo thoroughly.

The steps given here are not reproducible. A github issue of the same exists and points to the solution here. Following the solution we get this output:

The documentation does not explain what to do after this. There is a /input node which I am not sure if the rosbag should attach too or something else. Is there a ros bag that you guys have?

If I do ro2 node info /image_input_component this is what I get

I do not understand why the node is subscribing to the topic "/camera/camera_info" and publishing to "/camera/camera_info".

If I publish a rosbag to send data to "/camera/camera_info" and "/camera/image_raw". This happens:
image.png

This is due to the image_input_component node publishing and subscribing to the same camera_info. This leads to more camera_info messages coming than camera_rawImages. If you guys can clarify why this happens it would help a lot. Anyhow even with this working it does not seem like I am getting the result as shown in the github.

If you can let me know what the next steps are from here, it would help a lot. Thank you for the same

example not reproducible

Running the 'reproduction' section in the first benchmark example :

benchmarks/benchmarks/perception/a1_perception_2nodes/README.md

Line 25 in 609c1cd

cd ~/ros2_overlay_ws; colcon build --packages-select a1_perception_2nodes

yields

[1.034s] WARNING:colcon.colcon_core.package_selection:ignoring unknown package 'a1_perception_2nodes' in --packages-select
                     
Summary: 0 packages finished [0.43s]

There's no a1_perception_2nodes ROS2 package in the repository (no .xml files). Which is confirmed by running

benchmarks/benchmarks/perception/a1_perception_2nodes/README.md

Line 28 in 609c1cd

ros2 launch a1_perception_2nodes a1_perception_2nodes.launch.py

which gives

Package 'a1_perception_2nodes' not found: "package 'a1_perception_2nodes' not found, searching: ['/opt/ros/humble']"

In the following section, should lines 28 and 31 be the same? The comments state different explanations but the commands are the same.

benchmarks/benchmarks/perception/a1_perception_2nodes/README.md

Lines 27 to 31 in 609c1cd

 # Launch the benchmark 

 ros2 launch a1_perception_2nodes a1_perception_2nodes.launch.py 

 # Generate the report for the benchmark 

 ros2 launch a1_perception_2nodes a1_perception_2nodes.launch.py

Also, the a1_perception_2nodes.launch.py file is missing.

[NEW BENCHMARK] b1_visual_slam

Specify the ROS2 package are you benchmarking:
isaac_ros_visual_slam

Provide the link to the ROS2 package:
https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_visual_slam

Which category does your benchmark fall under?

What metric are you benchmarking?
Latency, throughput and power

June 23rd

What hardware will your reference implementation run on?
Workstation and AGX Orin

Your organization:
Acceleration Robotics

Your name:
Martiño Crespo Álvarez

Your email address or contact information:
[email protected]

[NEW BENCHMARK] depth_image_proc

Specify the ROS2 package are you benchmarking:
depth_image_proc

Provide the link to the ROS2 package:
https://github.com/ros-perception/image_pipeline/tree/humble/depth_image_proc

Which category does your benchmark fall under?

What metric are you benchmarking?
latency (ms)

What is your estimated date of completion?
May 2nd, 2023

What hardware will your reference implementation run on?

12th Gen Intel(R) Core(TM) i9-12900KF
NVIDIA GeForce RTX 3080 Ti

Your organization:
Harvard University

Your name:
Jason Jabbour

Your email address or contact information:
[email protected]

[NEW BENCHMARK] d6_direct_kinematics

Specify the ROS2 package are you benchmarking:
moveit2

Provide the link to the ROS2 package:
https://github.com/ros-planning/moveit2

Which category does your benchmark fall under?

What metric are you benchmarking?
Latency and power

What is your estimated date of completion?
July 28th, 2023

What hardware will your reference implementation run on?
Workstation

Your organization:
Acceleration Robotics

Your name:
Martiño Crespo Álvarez

Your email address or contact information:
[email protected]

[NEW BENCHMARK] d2_collision_checking_fcl

Specify the ROS2 package are you benchmarking:
moveit2

Provide the link to the ROS2 package:
https://github.com/ros-planning/moveit2

Which category does your benchmark fall under?

What metric are you benchmarking?
Latency and power

What is your estimated date of completion?
July 21th, 2023

What hardware will your reference implementation run on?
Workstation

Your organization:
Acceleration Robotics

Your name:
Martiño Crespo Álvarez

Your email address or contact information:
[email protected]

	# Launch the benchmark
	ros2 launch a1_perception_2nodes a1_perception_2nodes.launch.py

	# Generate the report for the benchmark
	ros2 launch a1_perception_2nodes a1_perception_2nodes.launch.py

robotperf / benchmarks Goto Github PK

benchmarks's People

Contributors

Stargazers

Watchers

Forkers

benchmarks's Issues

Additional information

Footnotes

Recommend Projects

Recommend Topics

Recommend Org