Thanks for all your work this project. I was able to get it running in a docker container the other day (using the dev_camera branch), and I'm really impressed with the render quality.

I noticed that RL is on your project roadmap - I'm curious to know what the current state of that is? I was playing around with it yesterday and came up with a gym-like loop, something like below:

# Start the simulation
self.timeline.play()

# The "infinite" loop
max_steps = 1000
t = 0
is_running = simulation_app.is_running()
while is_running and not self.stop_sim:

    if t >= max_steps:
        self.stop_sim = True

    is_running = simulation_app.is_running()

    # Update the UI of the app and perform the physics step
    self.world.step(render=True)

    if t % 30 == 0:
        x_c, y_c, z_c = self.camera.state['position']
        try:
            img = self.camera.state['frame'][:, :, :3]
            save_image(img, f'img/{t:03d}.png')
        except ValueError as e:
            print(f"[{t + 1}/{max_steps}] Could not generate image.\n{e}")

        print(f"[{t}] ({x_c:.3f}, {y_c:.3f}, {z_c:.3f}) - is_running: {is_running}")

    t += 1

# Cleanup and stop
carb.log_warn("PegasusApp Simulation App is closing.")
self.timeline.stop()
simulation_app.close()

I'm not sure of the best way to inject the actions though. Any thoughts on that? I was looking at the utils/nonlinear_controller example, and I'm thinking maybe that could be modded to work? Perhaps as simple as - in the continuous action case - setting the input_reference attribute with whatever values specified from the policy?

I've not had a lot of time to investigate this idea more deeply yet, but thought it would be useful to get your (and anyone else's interested) perspective on it before getting in too much deeper. I know you've got a lot to tackle on this project, so happy to submit a PR if we can get something working.

Hi @MarceloJacinto ,
Thanks for the amazing repository contributing to the world of drone simulation. I faced the following issue - after running example/3_ros_single_vehicle.py , I tried to run commander takeoff in px4 shell , and saw that it immediately switched on failsafe and disarmed. I tried with commander arm -f, and it followed the same. The launch threw warnings about the magnteometer , barometer and gps not present though the simulation was running. Is there anything wrong I'm doing? I also tried, ros2 run px4_ros_com offboard_launch , and nothing happened. What installation step am I missing or is the mavlink not connected to PX4 interface ?

When I use people api to create moving pedestrian(as shown in 9_people.py) in the isaac sim, I found that the mesh collider kept still no matter how human body moved. As a result, my lidar on carter robot could not detect moving pedestrian, but rgb camera and depth camera could.

pic1 is the simulation scene and pic2 displays the sensor of camera and lidar.

Not 100% sure if this is user error in the setup, but line 53 of the ros2_backend.py is assigning the value "3" to the rotor_callback's rotor_id. The result being all pubs to that topic control only rotor 3.

Hello,

My system details are:
Ubuntu 22.04,
ISAACSIM_PYTHON --version is 3.10.13
Isaac sim version is 2023.1.0-hotfix.1

I was following the instructions to install the pegasus extension. (Due to some issue on SIM I was unable to add the path to pegasus extensions folder, as a result, I pasted the pegasus extension folder in one of the existing paths, I was able to see it in extensions and also enable it as per the video)

After enabling the extension, I moved to the pip intsallation:

ISAACSIM_PYTHON -m pip install --editable pegasus.simulator

I get the error:

ISAACSIM_PYTHON -m pip install --editable pegasus.simulator
Obtaining file:///home/gvgampa/Documents/Kit/shared/exts/pegasus.simulator
Installing build dependencies ... done
Checking if build backend supports build_editable ... done
Getting requirements to build editable ... error
error: subprocess-exited-with-error

× Getting requirements to build editable did not run successfully.
│ exit code: 1
╰─> [19 lines of output]
Traceback (most recent call last):
File "/home/gvgampa/omniverse/Omniverse_lib/isaac_sim-2023.1.0-hotfix.1/kit/python/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in
main()
File "/home/gvgampa/omniverse/Omniverse_lib/isaac_sim-2023.1.0-hotfix.1/kit/python/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/home/gvgampa/omniverse/Omniverse_lib/isaac_sim-2023.1.0-hotfix.1/kit/python/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 132, in get_requires_for_build_editable
return hook(config_settings)
File "/tmp/pip-build-env-rtb6bnmt/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 441, in get_requires_for_build_editable
return self.get_requires_for_build_wheel(config_settings)
File "/tmp/pip-build-env-rtb6bnmt/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=['wheel'])
File "/tmp/pip-build-env-rtb6bnmt/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
self.run_setup()
File "/tmp/pip-build-env-rtb6bnmt/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 480, in run_setup
super().run_setup(setup_script=setup_script)
File "/tmp/pip-build-env-rtb6bnmt/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 311, in run_setup
exec(code, locals())
File "", line 8, in
ModuleNotFoundError: No module named 'toml'
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build editable did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
There was an error running python

Note that ISAACSIM_PYTHON -m pip install toml
results in
Requirement already satisfied: toml in /home/gvgampa/omniverse/Omniverse_lib/isaac_sim-2023.1.0-hotfix.1/kit/exts/omni.kit.pip_archive/pip_prebundle (0.10.1)

Furthermore when I launch ISAACSIM_PYTHON
I am able to import toml and use toml.load function.

If anyone has seen similar issue, please let me know how to resolve if possible. Thanks!

First of all, I would like to express my infinite appreciation for this project.
This project solved many of the test problems of PX4 and Isaac-sim that I have been curious about.
I found the following problems while installing the version of Isaac-sim-2023.1.0 according to the guide document.
I can't see Streaming Backend field in the Pegasus Simulator tab on the Isaac-sim-2023.1.0.
This is my screen capture.
I want to know what the problem is.

I look forward to your kind reply.

First of all I am very impressed with this new tool, thank you for sharing your work @MarceloJacinto.

Even tho the Documentation is excellent, I believe that the community would benefit from a "getting started" video tutorial, where installation as well as the basic functionalities are explored and explained. This would serve as a guide for new users, and make it more accessible for the general public.

Thanks again and keep up the good work!!

I understand this repository does not support ArduPilot and there appears to be nobody currently working on it.

I had started working on ArduPilot port using mavlink_backend.py but I am facing an issue where, my aircraft will move backwards after arming and takes off pitched backwards.

I have checked the following:

1. PWM values across all 4 rotors are equal upon takeoff during arming
2. Rotor direction is correct(compared the observed motion with PX4 implementation)

Initially i thought that,it would be a relatively simple for migrating from HIL messages from PX4 to ArduPilot's JSON SITL Interface, not sure if anyone has any suggestions to try or a direction to explore

Additional backend support for ArduPilot SITL, which is another trusted, versatile, and open source autopilot system.

• arducopter_launch_tool.py
• mavlink_backend.py: Remove hardcoded px4 references
• Be able to choose the wanted autopilot

Does the team have any leads on how to get depth_image from isaacsim? i was looking through the omini.issac documentation for camera sensor link.
i tried using their camera method

# inside rgb_camera.py file  
@Sensor.update_at_rate
def update(self, state: State, dt: float):
  self.camera.add_distance_to_camera_to_frame()
  # Get the most up to date frame from the camera
  data = self.camera.get_current_frame()
  depth = data.get('distance_to_camera', None)
  print(type(depth))

but the data returned is of NoneType for

Hello, When I use the UAV , the streaming Backend is only have PX4? Could you help me to solve it ?

Hello, firstly I want to thank the creators of the Pegasis Simulator. Currently, I am amidst the process of adding a custom controller; however, the drone is very erratic in nature. My intuition is that the drone's center of mass is off causing the applied force to mess with the drone's movement. Do you have any advice on what steps should be taken to implement a custom drone? For example, checking joint angles, pieces of code to change, etc. Anything would help

Firstly, fantastic work! This is a great project and will help the Px4 (drones) community working on Isaac Sim.

For offboard control, how easily can we use mavros (let's say with ROS1) for controlling the drone?

I see that you have a ROS2 backend (with some known issues), but nothing for ROS1.

I am interested in using mavros with ros1 for offboard control of the drone. Would this require me to custom make another backend?

First, thanks for sharing your great work.
I have referenced the 'RoadMap' in pegasuss docs for this simulator. According to the docs, there is currently no RGB camera in the list of supported sensors, so I would like to know when the camera sensor will be available in the simulator.
Also, I was wondering if it is Currently possible to communicate with object via Mavros based on ROS2 in its current state.

Thanks in advance.

Hi Marcelo,
I wanted to double check on this point. I am interested in controlling the drone using mavros in offboard mode and simulating it in isaac sim. Specifically, using attitude body rate setpoints and thrust.

The mavlink backend should be able to handle this right? It will make sure that the attitude setpoints sent via mavlink are tracked by px4 sitl, which in return will send the actuator controls for the rotors simulated in isaac sim.

Secondly, I noticed the mavlink backend sends ground truth information (but not position). I am wondering if that can have an impact on doing attitude setpoint control.

It seems that there's an error related to importing the tf2_ros module in the ros2_backend.py file of the Pegasus Simulator extension. Here's the log of isaac sim:

Isaac sim logs "Failed to import python module pegasus.simulator" when I switched on the PegasusSimulator extension in Isaac sim extension manager.

2024-04-16 08:03:49 [158,504ms] [Error] [omni.ext._impl.custom_importer] Failed to import the Python module pegasus.simulator. Error: No module named 'tf2_ros'. Traceback:
Traceback (most recent call last):
  File "/isaac-sim/kit/kernel/py/omni/ext/_impl/custom_importer.py", line 76, in import_module
    return importlib.import_module(name)
  File "/isaac-sim/kit/python/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/shared/codes/PegasusSimulator/extensions/pegasus.simulator/pegasus/simulator/__init__.py", line 9, in <module>
    from .extension import Pegasus_SimulatorExtension
  File "/shared/codes/PegasusSimulator/extensions/pegasus.simulator/pegasus/simulator/extension.py", line 31, in <module>
    from pegasus.simulator.ui.ui_window import WidgetWindow
  File "/shared/codes/PegasusSimulator/extensions/pegasus.simulator/pegasus/simulator/ui/__init__.py", line 5, in <module>
    from .ui_delegate import UIDelegate
  File "/shared/codes/PegasusSimulator/extensions/pegasus.simulator/pegasus/simulator/ui/ui_delegate.py", line 22, in <module>
    from pegasus.simulator.logic.backends import MavlinkBackend, MavlinkBackendConfig #, ROS2Backend
  File "/shared/codes/PegasusSimulator/extensions/pegasus.simulator/pegasus/simulator/logic/backends/__init__.py", line 8, in <module>
    from .ros2_backend import ROS2Backend
  File "/shared/codes/PegasusSimulator/extensions/pegasus.simulator/pegasus/simulator/logic/backends/ros2_backend.py", line 22, in <module>
    from tf2_ros.static_transform_broadcaster import StaticTransformBroadcaster
ModuleNotFoundError: No module named 'tf2_ros'

2024-04-16 08:03:49 [158,504ms] [Error] [carb.scripting-python.plugin] Exception: Extension Python module 'pegasus.simulator' in '/shared/codes/PegasusSimulator/extensions/pegasus.simulator' failed to load.

The import code in 'extensions/pegasus.simulator/pegasus/simulator/logic/backends/ros2_backend.py'

from tf2_ros.static_transform_broadcaster import StaticTransformBroadcaster
from tf2_ros.transform_broadcaster import TransformBroadcaster

causes this error, as the ROSBackend class imported in

whilst following the install instructions...

Ubuntu 20.04
IsaacSim 2023.1.0
ROS noetic installed

seems like the rclpy extension is not correctly setup as a dependency

[9525.730s] [ext: pegasus.simulator-1.0.0] startup
2023-12-07 20:07:28 [9,525,702ms] [Warning] [omni.isaac.core.utils.nucleus] /persistent/app/omniverse/mountedDrives setting not found
[9525.762s] [ext: omni.isaac.ros2_bridge-2.3.9] startup
2023-12-07 20:07:28 [9,525,772ms] [Error] [omni.isaac.ros2_bridge.scripts.extension] ROS_DISTRO of noetic is currently not supported
2023-12-07 20:07:28 [9,526,133ms] [Error] [omni.ext._impl.custom_importer] Failed to import python module pegasus.simulator. Error: No module named 'rclpy'. Traceback:
Traceback (most recent call last):
File "/home/duane/.local/share/ov/pkg/isaac_sim-2023.1.0-hotfix.1/kit/kernel/py/omni/ext/_impl/custom_importer.py", line 76, in import_module
return importlib.import_module(name)
File "/home/duane/.local/share/ov/pkg/isaac_sim-2023.1.0-hotfix.1/kit/python/lib/python3.10/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 1006, in _find_and_load_unlocked
File "", line 688, in _load_unlocked
File "", line 883, in exec_module
File "", line 241, in _call_with_frames_removed
File "/home/duane/PegasusSimulator/extensions/pegasus.simulator/pegasus/simulator/init.py", line 9, in
from .extension import Pegasus_SimulatorExtension
File "/home/duane/PegasusSimulator/extensions/pegasus.simulator/pegasus/simulator/extension.py", line 31, in
from pegasus.simulator.ui.ui_window import WidgetWindow
File "/home/duane/PegasusSimulator/extensions/pegasus.simulator/pegasus/simulator/ui/init.py", line 5, in
from .ui_delegate import UIDelegate
File "/home/duane/PegasusSimulator/extensions/pegasus.simulator/pegasus/simulator/ui/ui_delegate.py", line 22, in
from pegasus.simulator.logic.backends import MavlinkBackend, MavlinkBackendConfig #, ROS2Backend
File "/home/duane/PegasusSimulator/extensions/pegasus.simulator/pegasus/simulator/logic/backends/init.py", line 8, in
from .ros2_backend import ROS2Backend
File "/home/duane/PegasusSimulator/extensions/pegasus.simulator/pegasus/simulator/logic/backends/ros2_backend.py", line 16, in
import rclpy
ModuleNotFoundError: No module named 'rclpy'

2023-12-07 20:07:28 [9,526,133ms] [Error] [carb.scripting-python.plugin] Exception: Extension python module: 'pegasus.simulator' in '/home/duane/PegasusSimulator/extensions/pegasus.simulator' failed to load.

At:
/home/duane/.local/share/ov/pkg/isaac_sim-2023.1.0-hotfix.1/kit/kernel/py/omni/ext/_impl/_internal.py(213): startup
/home/duane/.local/share/ov/pkg/isaac_sim-2023.1.0-hotfix.1/kit/kernel/py/omni/ext/_impl/_internal.py(328): startup_extension
PythonExtension.cpp::startup()(2):
/home/duane/.local/share/ov/pkg/isaac_sim-2023.1.0-hotfix.1/kit/exts/omni.kit.window.extensions/omni/kit/window/extensions/ext_commands.py(29): do
/home/duane/.local/share/ov/pkg/isaac_sim-2023.1.0-hotfix.1/kit/exts/omni.kit.commands/omni/kit/undo/undo.py(80): execute
/home/duane/.local/share/ov/pkg/isaac_sim-2023.1.0-hotfix.1/kit/exts/omni.kit.commands/omni/kit/commands/command.py(463): execute
/home/duane/.local/share/ov/pkg/isaac_sim-2023.1.0-hotfix.1/kit/exts/omni.kit.window.extensions/omni/kit/window/extensions/common.py(288): toggle_extension
/home/duane/.local/share/ov/pkg/isaac_sim-2023.1.0-hotfix.1/kit/exts/omni.kit.window.extensions/omni/kit/window/extensions/ext_components.py(93): toggle

2023-12-07 20:07:28 [9,526,133ms] [Error] [omni.ext.plugin] [ext: pegasus.simulator-1.0.0] Failed to startup python extension.
[9526.884s] [ext: omni.isaac.ros2_bridge-2.3.9] shutdown

I'm eager to use ROS 2 with Isaac Sim and Pegasus, but haven't found any official guides to get started. Px4 Mavlink seems to be the default integration, but are there resources available for using ROS 2 with Pegasus? Currently, I only see Px4 options within the Isaac Sim panel.
Thanks in advance.

I followed the installation instructions with Ubuntu 22.04 however when I click load scene I am getting the following output in the terminal and it cannot load.

2024-01-06 13:22:34 [186,398ms] [Error] [asyncio] Task exception was never retrieved future: <Task finished name='Task-333' coro=<PegasusInterface.load_environment_async() done, defined at /home/volo/PegasusSimulator/extensions/pegasus.simulator/pegasus/simulator/logic/interface/pegasus_interface.py:220> exception=AttributeError("'NoneType' object has no attribute 'warm_start'")> Traceback (most recent call last): File "/home/volo/PegasusSimulator/extensions/pegasus.simulator/pegasus/simulator/logic/interface/pegasus_interface.py", line 231, in load_environment_async await self.world.reset_async() File "/home/volo/.local/share/ov/pkg/isaac_sim-2023.1.1/exts/omni.isaac.core/omni/isaac/core/world/world.py", line 520, in reset_async await self.reset_async_no_set_up_scene(soft=soft) File "/home/volo/.local/share/ov/pkg/isaac_sim-2023.1.1/exts/omni.isaac.core/omni/isaac/core/world/world.py", line 480, in reset_async_no_set_up_scene await SimulationContext.reset_async(self, soft=soft) File "/home/volo/.local/share/ov/pkg/isaac_sim-2023.1.1/exts/omni.isaac.core/omni/isaac/core/simulation_context/simulation_context.py", line 653, in reset_async await self.play_async() File "/home/volo/.local/share/ov/pkg/isaac_sim-2023.1.1/exts/omni.isaac.core/omni/isaac/core/simulation_context/simulation_context.py", line 869, in play_async self.get_physics_context().warm_start() AttributeError: 'NoneType' object has no attribute 'warm_start'

Additionally I also cannot proceed with the installation after the pip command. I am getting following error:

Run the pip command using the built-in python interpreter
ISAACSIM_PYTHON -m pip install --editable pegasus.simulator

ERROR: Command errored out with exit status 1: command: /home/volo/.local/share/ov/pkg/isaac_sim-2023.1.1/kit/python/bin/python3 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/volo/PegasusSimulator/extensions/pegasus.simulator/setup.py'"'"'; __file__='"'"'/home/volo/PegasusSimulator/extensions/pegasus.simulator/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-q0fvb8w7 cwd: /home/volo/PegasusSimulator/extensions/pegasus.simulator/ Complete output (1 lines): error in pegasus-simulator setup command: 'python_requires' must be a string containing valid version specifiers; Invalid specifier: '>=3.7.*' ---------------------------------------- WARNING: Discarding file:///home/volo/PegasusSimulator/extensions/pegasus.simulator. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output. ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output. WARNING: You are using pip version 21.2.1+nv1; however, version 23.3.2 is available. You should consider upgrading via the '/home/volo/.local/share/ov/pkg/isaac_sim-2023.1.1/kit/python/bin/python3 -m pip install --upgrade pip' command. There was an error running python

Thanks in advance for the help!