haosulab / maniskill Goto Github PK

SAPIEN Manipulation Skill Framework, a GPU parallelized robotics simulator and benchmark

Home Page: https://maniskill.readthedocs.io/en/latest/

License: Apache License 2.0

Dockerfile 0.14% Python 99.68% Shell 0.15% CMake 0.03%

3d-computer-vision reinforcement-learning robotics computer-vision robot-manipulation simulation-environment embodied-ai robotics-simulation

maniskill's Introduction

ManiSkill 3 (Beta)

Sample of environments/robots rendered with ray-tracing. Scene datasets sourced from AI2THOR and ReplicaCAD

ManiSkill is a powerful unified framework for robot simulation and training powered by SAPIEN. The entire stack is as open-source as possible and ManiSkill v3 is in beta release now. Among its features include:

GPU parallelized visual data collection system. On the high end you can collect RGBD + Segmentation data at 20k FPS with a 4090 GPU, 10-100x faster compared to most other simulators.
Example tasks covering a wide range of different robot embodiments (quadruped, mobile manipulators, single-arm robots) as well as a wide range of different tasks (table-top, locomotion, dextrous manipulation)
GPU parallelized tasks, enabling incredibly fast synthetic data collection in simulation
GPU parallelized tasks support simulating diverse scenes where every parallel environment has a completely different scene/set of objects
Flexible task building API that abstracts away much of the complex GPU memory management code

ManiSkill plans to enable all kinds of workflows, including but not limited to 2D/3D vision-based reinforcement learning, imitation learning, sense-plan-act, etc. There will also be more assets/scenes supported by ManiSkill (e.g. AI2THOR) in addition to other features such as digital-twins for evaluation of real-world policies, see our roadmap for planned features that will be added over time before the official v3 is released.

Please refer to our documentation to learn more information from tutorials on building tasks to data collection.

NOTE: This project currently is in a beta release, so not all features have been added in yet and there may be some bugs. If you find any bugs or have any feature requests please post them to our GitHub issues or discuss about them on GitHub discussions. We also have a Discord Server through which we make announcements and discuss about ManiSkill.

Users looking for the original ManiSkill2 can find the commit for that codebase at the v0.5.3 tag

Installation

Installation of ManiSkill is extremely simple, you only need to run a few pip installs

# install the package
pip install --upgrade mani_skill
# install a version of torch that is compatible with your system
pip install torch torchvision torchaudio

Finally you also need to set up Vulkan with instructions here

For more details about installation (e.g. from source, or doing troubleshooting) see the documentation

Getting Started

To get started, check out the quick start documentation: https://maniskill.readthedocs.io/en/latest/user_guide/getting_started/quickstart.html

We also have a quick start colab notebook that lets you try out GPU parallelized simulation without needing your own hardware. Everything is runnable on Colab free tier.

For a full list of example scripts you can run see the docs.

License

All rigid body environments in ManiSkill are licensed under fully permissive licenses (e.g., Apache-2.0).

The assets are licensed under CC BY-NC 4.0.

maniskill's People

Contributors

Stargazers

Watchers

maniskill's Issues

RuntimeError: createInstance: No Vulkan extensions found for window surface creation (hint: set VK_ICD_FILENAMES to `locate icd.json`)

I think I followed the instructions in the github page step by step but encounter this problem. Is there any way to solve the problem? Sorry that I'm new to Vulkan.

mani_skill2 - ERROR - 'NoneType' object has no attribute 'vertices'

Hello!

Since there's some current issue with Google Colab (https://github.com/haosulab/ManiSkill2/issues/85), I decided to switch to a gpu server (ssh from my local mac, I'm using XQuartz and X11 forwarding)

My test jupyter notebook seems to render the env fine

import gym
from tqdm.notebook import tqdm
import numpy as np
import mani_skill2.envs
import matplotlib.pyplot as plt

env_id = "OpenCabinetDrawer-v1"
obs_mode = "rgbd" #@param can be one of ['pointcloud', 'rgbd', 'state_dict', 'state']
# choose a controller type / action space, see https://haosulab.github.io/ManiSkill2/concepts/controllers.html for a full list
control_mode = "base_pd_joint_vel_arm_pd_joint_vel"

reward_mode = "dense" #@param can be one of ['sparse', 'dense']


env = gym.make(env_id, obs_mode=obs_mode, reward_mode=reward_mode, control_mode=control_mode)
obs = env.reset()
print("Action Space:", env.action_space)
# take a look at the current state
img = env.render(mode="cameras")
plt.figure(figsize=(10,6))
plt.title("Current State")
plt.imshow(img)
env.close()

But when I tried th VecEnv,

from mani_skill2.vector import VecEnv, make as make_vec_env
num_envs = 2 # recommended value for Google Colab. If you have more cores and a more powerful GPU you can increase this
env = make_vec_env(
    env_id,
    num_envs,
    obs_mode=obs_mode,
    reward_mode=reward_mode,
    control_mode=control_mode,
)
env.seed(0)
obs = env.reset()
print("Base Camera RGB:", obs['image']['base_camera']['rgb'].shape)
print("Base Camera RGB device:", obs['image']['base_camera']['rgb'].device)
env.close()

I'm getting these errors

I'm wondering if it might be related to https://github.com/haosulab/ManiSkill2/issues/28. Unfortunately, I don't have sudo privileges in this gpu server so I can only manage my local setup using conda and pip.

[2023-05-01 21:46:51.371] [svulkan2] [warning] Only 1 renderer is allowed per process. All previously created renderer resources are now invalid

2023-05-01 21:46:51,609 - mani_skill2 - INFO - RenderServer is running at: localhost:41705
2023-05-01 21:47:15,926 - mani_skill2 - ERROR - 'NoneType' object has no attribute 'vertices'
Traceback (most recent call last):
  File "/home/uname/RobotLearning/ManiSkill2/mani_skill2/vector/vec_env.py", line 56, in _worker
    env = env_fn()
  File "/home/uname/RobotLearning/ManiSkill2/mani_skill2/vector/registration.py", line 11, in _make_env
    env = env_spec.make(**kwargs)
  File "/home/uname/RobotLearning/ManiSkill2/mani_skill2/utils/registration.py", line 34, in make
    return self.cls(**_kwargs)
  File "/home/uname/RobotLearning/ManiSkill2/mani_skill2/envs/ms1/open_cabinet_door_drawer.py", line 38, in __init__
    super().__init__(*args, **kwargs)
  File "/home/uname/RobotLearning/ManiSkill2/mani_skill2/envs/ms1/base_env.py", line 55, in __init__
    super().__init__(*args, **kwargs)
  File "/home/uname/RobotLearning/ManiSkill2/mani_skill2/envs/sapien_env.py", line 178, in __init__
    obs = self.reset(reconfigure=True)
  File "/home/uname/RobotLearning/ManiSkill2/mani_skill2/envs/ms1/open_cabinet_door_drawer.py", line 159, in reset
    return super().reset(seed=seed, reconfigure=reconfigure, model_id=model_id)
  File "/home/uname/RobotLearning/ManiSkill2/mani_skill2/envs/ms1/base_env.py", line 87, in reset
    ret = super().reset(seed=self._episode_seed, reconfigure=reconfigure)
  File "/home/uname/RobotLearning/ManiSkill2/mani_skill2/envs/sapien_env.py", line 473, in reset
    self.reconfigure()
  File "/home/uname/RobotLearning/ManiSkill2/mani_skill2/envs/sapien_env.py", line 359, in reconfigure
    self._load_articulations()
  File "/home/uname/RobotLearning/ManiSkill2/mani_skill2/envs/ms1/open_cabinet_door_drawer.py", line 66, in _load_articulations
    self._set_cabinet_handles_mesh()
  File "/home/uname/RobotLearning/ManiSkill2/mani_skill2/envs/ms1/open_cabinet_door_drawer.py", line 94, in _set_cabinet_handles_mesh
    meshes.extend(get_visual_body_meshes(visual_body))
  File "/home/uname/RobotLearning/ManiSkill2/mani_skill2/utils/trimesh_utils.py", line 40, in get_visual_body_meshes
    vertices = render_shape.mesh.vertices * visual_body.scale  # [n, 3]
AttributeError: 'NoneType' object has no attribute 'vertices'
2023-05-01 21:47:15,927 - mani_skill2 - ERROR - 'NoneType' object has no attribute 'vertices'
Traceback (most recent call last):
  File "/home/uname/RobotLearning/ManiSkill2/mani_skill2/vector/vec_env.py", line 56, in _worker
    env = env_fn()
  File "/home/uname/RobotLearning/ManiSkill2/mani_skill2/vector/registration.py", line 11, in _make_env
    env = env_spec.make(**kwargs)
  File "/home/uname/RobotLearning/ManiSkill2/mani_skill2/utils/registration.py", line 34, in make
    return self.cls(**_kwargs)
  File "/home/uname/RobotLearning/ManiSkill2/mani_skill2/envs/ms1/open_cabinet_door_drawer.py", line 38, in __init__
    super().__init__(*args, **kwargs)
  File "/home/uname/RobotLearning/ManiSkill2/mani_skill2/envs/ms1/base_env.py", line 55, in __init__
    super().__init__(*args, **kwargs)
  File "/home/uname/RobotLearning/ManiSkill2/mani_skill2/envs/sapien_env.py", line 178, in __init__
    obs = self.reset(reconfigure=True)
  File "/home/uname/RobotLearning/ManiSkill2/mani_skill2/envs/ms1/open_cabinet_door_drawer.py", line 159, in reset
    return super().reset(seed=seed, reconfigure=reconfigure, model_id=model_id)
  File "/home/uname/RobotLearning/ManiSkill2/mani_skill2/envs/ms1/base_env.py", line 87, in reset
    ret = super().reset(seed=self._episode_seed, reconfigure=reconfigure)
  File "/home/uname/RobotLearning/ManiSkill2/mani_skill2/envs/sapien_env.py", line 473, in reset
    self.reconfigure()
  File "/home/uname/RobotLearning/ManiSkill2/mani_skill2/envs/sapien_env.py", line 359, in reconfigure
    self._load_articulations()
  File "/home/uname/RobotLearning/ManiSkill2/mani_skill2/envs/ms1/open_cabinet_door_drawer.py", line 66, in _load_articulations
    self._set_cabinet_handles_mesh()
  File "/home/uname/RobotLearning/ManiSkill2/mani_skill2/envs/ms1/open_cabinet_door_drawer.py", line 94, in _set_cabinet_handles_mesh
    meshes.extend(get_visual_body_meshes(visual_body))
  File "/home/uname/RobotLearning/ManiSkill2/mani_skill2/utils/trimesh_utils.py", line 40, in get_visual_body_meshes
    vertices = render_shape.mesh.vertices * visual_body.scale  # [n, 3]
AttributeError: 'NoneType' object has no attribute 'vertices'

When replaying: "Episode XYZ is not replayed successfully. Skipping"

Is it a concern if we see Episode XYZ is not replayed successfully. Skipping when we replay trajectories? I am following the README https://github.com/haosulab/ManiSkill2#demonstrations.

Here is an example with me loading PlugCharger-v0. The command is based directly from the README:

python tools/replay_trajectory.py --traj-path demos/rigid_body_envs/PlugCharger-v0/trajectory.h5  \
    --save-traj --target-control-mode pd_joint_delta_pos --num-procs 10

Here is my command line output:

(mani_skill2) seita@takeshi:~/ManiSkill2 (main)$ python tools/replay_trajectory.py --traj-path demos/rigid_body_envs/PlugCharger-v0/trajectory.h5  --save-traj --target-control-mode pd_joint_delta_pos --num-procs 10
0step [00:00, ?step/s][2022-08-09 10:05:45.243] [SAPIEN] [warning] A second engine will share the same internal structures with the first one. Arguments passed to constructor will be ignored.
[2022-08-09 10:05:45.244] [SAPIEN] [warning] A second renderer will share the same internal context with the first one. Arguments passed to constructor will be ignored.
[2022-08-09 10:05:45.244] [SAPIEN] [warning] Setting renderer more than once should be avoided.
0step [00:00, ?step/s, control_mode=pd_joint_delta_pos, obs_mode=none][2022-08-09 10:05:45.321] [SAPIEN] [warning] A second engine will share the same internal structures with the first one. Arguments passed to constructor will be ignored.
[2022-08-09 10:05:45.321] [SAPIEN] [warning] A second renderer will share the same internal context with the first one. Arguments passed to constructor will be ignored.
[2022-08-09 10:05:45.321] [SAPIEN] [warning] Setting renderer more than once should be avoided.
Replaying traj_0:   7%|█▊                         | 10/147 [00:00<00:01, 95.58step/s, control_mode=pd_joint_delta_pos, obs_mode=none[2022-08-09 10:05:45.546] [SAPIEN] [warning] A second engine will share the same internal structures with the first one. Arguments passed to constructor will be ignored.
[2022-08-09 10:05:45.546] [SAPIEN] [warning] A second renderer will share the same internal context with the first one. Arguments passed to constructor will be ignored.
[2022-08-09 10:05:45.546] [SAPIEN] [warning] Setting renderer more than once should be avoided.
Replaying traj_0:  14%|███▋                       | 20/147 [00:00<00:01, 95.15step/s, control_mode=pd_joint_delta_pos, obs_mode=none][2022-08-09 10:05:45.562] [SAPIEN] [warning] A second engine will share the same internal structures with the first one. Arguments passed to constructor will be ignored.
[2022-08-09 10:05:45.562] [SAPIEN] [warning] A second renderer will share the same internal context with the first one. Arguments passed to constructor will be ignored.
[2022-08-09 10:05:45.562] [SAPIEN] [warning] Setting renderer more than once should be avoided.
[2022-08-09 10:05:45.597] [SAPIEN] [warning] A second engine will share the same internal structures with the first one. Arguments passed to constructor will be ignored.
[2022-08-09 10:05:45.597] [SAPIEN] [warning] A second renderer will share the same internal context with the first one. Arguments passed to constructor will be ignored.
[2022-08-09 10:05:45.597] [SAPIEN] [warning] Setting renderer more than once should be avoided.
                                                                      [2022-08-09 10:05:45.663] [SAPIEN] [warning] A second engine will share the same internal structures with the first one. Arguments passed to constructor will be ignored.
[2022-08-09 10:05:45.663] [SAPIEN] [warning] A second renderer will share the same internal context with the first one. Arguments passed to constructor will be ignored.=pd_joint_delta_pos, obs_mode=none]
[2022-08-09 10:05:45.663] [SAPIEN] [warning] Setting renderer more than once should be avoided.
[2022-08-09 10:05:45.666] [SAPIEN] [warning] A second engine will share the same internal structures with the first one. Arguments passed to constructor will be ignored.pd_joint_delta_pos, obs_mode=none]
[2022-08-09 10:05:45.666] [SAPIEN] [warning] A second renderer will share the same internal context with the first one. Arguments passed to constructor will be ignored.
[2022-08-09 10:05:45.666] [SAPIEN] [warning] Setting renderer more than once should be avoided.
[2022-08-09 10:05:45.684] [SAPIEN] [warning] A second engine will share the same internal structures with the first one. Arguments passed to constructor will be ignored.
[2022-08-09 10:05:45.684] [SAPIEN] [warning] A second renderer will share the same internal context with the first one. Arguments passed to constructor will be ignored.
[2022-08-09 10:05:45.684] [SAPIEN] [warning] Setting renderer more than once should be avoided.
Replaying traj_0:  20%|█████▌                     | 30/147 [00:00<00:01, 86.33step/s, contro[2022-08-09 10:05:45.712] [SAPIEN] [warning] A second engine will share the same internal structures with the first one. Arguments passed to constructor will be ignored.none]
[2022-08-09 10:05:45.712] [SAPIEN] [warning] A second renderer will share the same internal context with the first one. Arguments passed to constructor will be ignored. ?step/s, control_mode=pd_joint_delta_pos, obs_mode=none]
[2022-08-09 10:05:45.712] [SAPIEN] [warning] Setting renderer more than once should be avoided.
                                                                                            [2022-08-09 10:05:45.735] [SAPIEN] [warning] A second engine will share the same internal structures with the first one. Arguments passed to constructor will be ignored.
[2022-08-09 10:05:45.735] [SAPIEN] [warning] A second renderer will share the same internal context with the first one. Arguments passed to constructor will be ignored.                        | 0/147 [00:00<?, ?step/s, control_mode=pd_joint_delta_pos, obs_mode=none]
[2022-08-09 10:05:45.735] [SAPIEN] [warning] Setting renderer more than once should be avoided.
Episode 504 is not replayed successfully. Skipping 155/178 [00:01<00:00, 59.01step/s, control_mode=pd_joint_delta_pos, obs_mode=none]
Episode 521 is not replayed successfully. Skipping 83/152 [00:00<00:00, 107.15step/s, control_mode=pd_joint_delta_pos, obs_mode=none]
Episode 727 is not replayed successfully. Skipping| 83/158 [00:00<00:00, 95.67step/s, control_mode=pd_joint_delta_pos, obs_mode=none]
Episode 333 is not replayed successfully. Skipping 15/197 [00:00<00:01, 142.84step/s, control_mode=pd_joint_delta_pos, obs_mode=none]
Episode 338 is not replayed successfully. Skipping 143/153 [00:01<00:00, 57.13step/s, control_mode=pd_joint_delta_pos, obs_mode=none]
Episode 40 is not replayed successfully. Skipping                                                                                    
Episode 143 is not replayed successfully. Skipping 133/143 [00:01<00:00, 53.05step/s, control_mode=pd_joint_delta_pos, obs_mode=none]
Episode 346 is not replayed successfully. Skipping| 87/132 [00:01<00:00, 67.90step/s, control_mode=pd_joint_delta_pos, obs_mode=none]
Episode 247 is not replayed successfully. Skipping| 95/132 [00:01<00:00, 63.21step/s, control_mode=pd_joint_delta_pos, obs_mode=none]
Episode 150 is not replayed successfully. Skipping 108/161 [00:01<00:00, 61.83step/s, control_mode=pd_joint_delta_pos, obs_mode=none]
Episode 750 is not replayed successfully. Skipping 28/168 [00:00<00:01, 136.35step/s, control_mode=pd_joint_delta_pos, obs_mode=none]
Episode 452 is not replayed successfully. Skipping 104/131 [00:01<00:00, 60.44step/s, control_mode=pd_joint_delta_pos, obs_mode=none]
Episode 553 is not replayed successfully. Skipping 153/181 [00:02<00:00, 53.43step/s, control_mode=pd_joint_delta_pos, obs_mode=none]
Episode 554 is not replayed successfully. Skipping 135/142 [00:02<00:00, 48.13step/s, control_mode=pd_joint_delta_pos, obs_mode=none]
Episode 457 is not replayed successfully. Skipping 114/147 [00:01<00:00, 62.83step/s, control_mode=pd_joint_delta_pos, obs_mode=none]
Episode 858 is not replayed successfully. Skipping 115/139 [00:01<00:00, 58.61step/s, control_mode=pd_joint_delta_pos, obs_mode=none]
Episode 960 is not replayed successfully. Skipping| 80/144 [00:00<00:00, 78.27step/s, control_mode=pd_joint_delta_pos, obs_mode=none]
Episode 558 is not replayed successfully. Skipping 70/163 [00:00<00:00, 121.83step/s, control_mode=pd_joint_delta_pos, obs_mode=none]
Episode 360 is not replayed successfully. Skipping| 71/147 [00:00<00:00, 88.15step/s, control_mode=pd_joint_delta_pos, obs_mode=none]
Episode 864 is not replayed successfully. Skipping| 87/144 [00:00<00:00, 69.19step/s, control_mode=pd_joint_delta_pos, obs_mode=none]
Episode 265 is not replayed successfully. Skipping 102/144 [00:01<00:00, 60.93step/s, control_mode=pd_joint_delta_pos, obs_mode=none]
Episode 870 is not replayed successfully. Skipping 155/184 [00:01<00:00, 57.15step/s, control_mode=pd_joint_delta_pos, obs_mode=none]
Episode 272 is not replayed successfully. Skipping| 81/147 [00:00<00:00, 96.98step/s, control_mode=pd_joint_delta_pos, obs_mode=none]
Episode 177 is not replayed successfully. Skipping 115/147 [00:01<00:00, 61.02step/s, control_mode=pd_joint_delta_pos, obs_mode=none]
Episode 381 is not replayed successfully. Skipping| 89/137 [00:00<00:00, 69.99step/s, control_mode=pd_joint_delta_pos, obs_mode=none]
Episode 682 is not replayed successfully. Skipping 124/137 [00:01<00:00, 55.26step/s, control_mode=pd_joint_delta_pos, obs_mode=none]
Episode 87 is not replayed successfully. Skipping                                                                                    
Episode 690 is not replayed successfully. Skipping 42/176 [00:00<00:01, 132.18step/s, control_mode=pd_joint_delta_pos, obs_mode=none]
Episode 393 is not replayed successfully. Skipping 135/178 [00:01<00:00, 59.71step/s, control_mode=pd_joint_delta_pos, obs_mode=none]
Episode 998 is not replayed successfully. Skipping 44/178 [00:00<00:01, 126.21step/s, control_mode=pd_joint_delta_pos, obs_mode=none]
Replaying traj_99: 100%|█████████████████████████| 178/178 [00:02<00:00, 68.15step/s, control_mode=pd_joint_delta_pos, obs_mode=none]
Episode 699 is not replayed successfully. Skipping 139/196 [00:01<00:00, 61.29step/s, control_mode=pd_joint_delta_pos, obs_mode=none]
Episode 598 is not replayed successfully. Skipping                                                                                   
Episode 399 is not replayed successfully. Skipping                                                                                   
Replaying traj_398:   0%|                                  | 0/162 [00:00<?, ?step/s, control_mode=pd_joint_delta_pos, obs_mode=noneMerge to demos/rigid_body_envs/PlugCharger-v0/trajectory.none.pd_joint_delta_pos.h5/s, control_mode=pd_joint_delta_pos, obs_mode=none]
Merging demos/rigid_body_envs/PlugCharger-v0/trajectory.none.pd_joint_delta_pos.0.h5                                                 
Merging demos/rigid_body_envs/PlugCharger-v0/trajectory.none.pd_joint_delta_pos.1.h5                                                 
Merging demos/rigid_body_envs/PlugCharger-v0/trajectory.none.pd_joint_delta_pos.2.h5, control_mode=pd_joint_delta_pos, obs_mode=none]
Merging demos/rigid_body_envs/PlugCharger-v0/trajectory.none.pd_joint_delta_pos.3.h5                                                 
Merging demos/rigid_body_envs/PlugCharger-v0/trajectory.none.pd_joint_delta_pos.4.h5, control_mode=pd_joint_delta_pos, obs_mode=none]
Merging demos/rigid_body_envs/PlugCharger-v0/trajectory.none.pd_joint_delta_pos.5.h5, control_mode=pd_joint_delta_pos, obs_mode=none]
Merging demos/rigid_body_envs/PlugCharger-v0/trajectory.none.pd_joint_delta_pos.6.h5, control_mode=pd_joint_delta_pos, obs_mode=none]
Merging demos/rigid_body_envs/PlugCharger-v0/trajectory.none.pd_joint_delta_pos.7.h5, control_mode=pd_joint_delta_pos, obs_mode=none]
Merging demos/rigid_body_envs/PlugCharger-v0/trajectory.none.pd_joint_delta_pos.8.h5
Merging demos/rigid_body_envs/PlugCharger-v0/trajectory.none.pd_joint_delta_pos.9.h5
Remove demos/rigid_body_envs/PlugCharger-v0/trajectory.none.pd_joint_delta_pos.0.h5
Remove demos/rigid_body_envs/PlugCharger-v0/trajectory.none.pd_joint_delta_pos.0.json
Remove demos/rigid_body_envs/PlugCharger-v0/trajectory.none.pd_joint_delta_pos.1.h5
Remove demos/rigid_body_envs/PlugCharger-v0/trajectory.none.pd_joint_delta_pos.1.json
Remove demos/rigid_body_envs/PlugCharger-v0/trajectory.none.pd_joint_delta_pos.2.h5
Remove demos/rigid_body_envs/PlugCharger-v0/trajectory.none.pd_joint_delta_pos.2.json
Remove demos/rigid_body_envs/PlugCharger-v0/trajectory.none.pd_joint_delta_pos.3.h5
Remove demos/rigid_body_envs/PlugCharger-v0/trajectory.none.pd_joint_delta_pos.3.json
Remove demos/rigid_body_envs/PlugCharger-v0/trajectory.none.pd_joint_delta_pos.4.h5
Remove demos/rigid_body_envs/PlugCharger-v0/trajectory.none.pd_joint_delta_pos.4.json
Remove demos/rigid_body_envs/PlugCharger-v0/trajectory.none.pd_joint_delta_pos.5.h5
Remove demos/rigid_body_envs/PlugCharger-v0/trajectory.none.pd_joint_delta_pos.5.json
Remove demos/rigid_body_envs/PlugCharger-v0/trajectory.none.pd_joint_delta_pos.6.h5
Remove demos/rigid_body_envs/PlugCharger-v0/trajectory.none.pd_joint_delta_pos.6.json
Remove demos/rigid_body_envs/PlugCharger-v0/trajectory.none.pd_joint_delta_pos.7.h5
Remove demos/rigid_body_envs/PlugCharger-v0/trajectory.none.pd_joint_delta_pos.7.json
Remove demos/rigid_body_envs/PlugCharger-v0/trajectory.none.pd_joint_delta_pos.8.h5
Remove demos/rigid_body_envs/PlugCharger-v0/trajectory.none.pd_joint_delta_pos.8.json
Remove demos/rigid_body_envs/PlugCharger-v0/trajectory.none.pd_joint_delta_pos.9.h5
Remove demos/rigid_body_envs/PlugCharger-v0/trajectory.none.pd_joint_delta_pos.9.json
(mani_skill2) seita@takeshi:~/ManiSkill2 (main)$

As you can see there are a lot of episodes that did not seem to be replayed successfully. It happens in these lines of code:

https://github.com/haosulab/ManiSkill2/blob/0bac111d0750126d7f3521fd7a9c04d68518f14b/tools/replay_trajectory.py#L334-L384

To clarify, the script stills create the expected .h5 and .json files in the appropriate directory:

(mani_skill2) seita@takeshi:~/ManiSkill2 (main)$ ls -lh demos/rigid_body_envs/PlugCharger-v0/
total 99M
-rw-rw-r-- 1 seita seita  50M Aug  4 11:27 trajectory.h5
-rw-rw-r-- 1 seita seita 357K Aug  4 11:27 trajectory.json
-rw-rw-r-- 1 seita seita  48M Aug  9 10:09 trajectory.none.pd_joint_delta_pos.h5
-rw-rw-r-- 1 seita seita 351K Aug  9 10:09 trajectory.none.pd_joint_delta_pos.json
(mani_skill2) seita@takeshi:~/ManiSkill2 (main)$

But I want to be sure that it is OK to see these error messages when we replay the trajectory. (If so, I am also wondering why this occurs.)

I do not see this error when loading: PickCube-v0, StackCube-v0. When I load PegInsertionSide-v0 it only has one unsuccessful replay, episode 786. In fact if I try this again, 786 is still the only one that fails, suggesting that this is deterministic.

Errors while creating submissions

Error 1

When I followed the Participation Guidelines, like:

# Add your submission to PYTHONPATH. Ensure that "user_solution.py" can be found to import.
export PYTHONPATH=${PATH_TO_YOUR_CODES_IN_HOST}:$PYTHONPATH

# Test whether the user solution can be imported
# python -c "from user_solution import UserPolicy"

# Run evaluation. The result will be saved to ${OUTPUT_DIR}.
ENV_ID="PickCube-v0" OUTPUT_DIR="tmp"
python -m mani_skill2.evaluation.run_evaluation -e ${ENV_ID} -o ${OUTPUT_DIR}

The path of my user_solution.py is /data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-main/submission
So I add my submission to PYTHONPATH and test whether the user solution can be imported like this:

export PYTHONPATH=/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-main/submission:$PYTHONPATH
python -c "from user_solution import UserPolicy"

The error comes out like this:

Pointnet++ is not compiled
No module named 'maniskill2_learn.networks.modules.pn2_modules'

Although after I installed Pointnet++, this error still exists.

Error 2

When I tried to run a local evaluation like this:

ENV_ID="PickCube-v0" OUTPUT_DIR="tmp"
python -m mani_skill2.evaluation.run_evaluation -e ${ENV_ID} -o ${OUTPUT_DIR}

The error comes out like this:

Fail to load evaluation configuration. (<class 'TypeError'>, TypeError("'NoneType' object cannot be interpreted as an integer"))

Error 3

When I built my submission docker image like this:

docker build -f "/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-main/Docker" -t maniskill2023-submission

The error comes out like this:

ERROR: "docker buildx build" requires exactly 1 argument.
See 'docker buildx build --help'.

Usage:  docker buildx build [OPTIONS] PATH | URL | -

Start a build

I've been trying to figure out how to fix those errors for a couple of days, but I've failed. I also can't tell if${YOUR_CODES_AND_WEIGHTS_IN_CONTAINER}is same as ${YOUR_CODES_AND_WEIGHTS_IN_HOST} since I haven't found an explanation for ${YOUR_CODES_AND_WEIGHTS_IN_CONTAINER} in the guideline.
It would be fantastic if you could assist me in resolving those errors and provide us with more detailed Participation Guidelines. Your kind assistance is much appreciated!

The score on leaderboard is 0 under 'env_name=OpenCabinetDoor-v1'

Hi, When I got a .ckpt file after I trained locally, the success rate of it could be evaluated at nearly 0.2. The submission docker can be tested normally, but when I uploaded it to the leaderboard, the score showed 0 :

I had double checked user_solution.py on e.g., controller and observation mode, and used the latest ManiSkill2-learn. However, the scores were still 0, and it seems that the leaderboard did not fully utilize the dataset:

but even so, I don't think the score of OpenCabinetDoor-v1/train could be 0. What could be causing this phenomenon?

Support of granular object task

Thanks for providing so many interesting and challenging tasks for the rigid body and soft body! I'm wondering if the simulator could support testing granular objects, e.g. sand, coffee beans, or nuts. If so, what is the possible advantage and disadvantage of the ManiSkill2 simulator compared with others, like PyFlex, Isaac gym, or Tachi-MPM?

More objects of partnet_mobility

In this page, you released 10 thousand partet mobility objects. However, the python tools/download.py --uid faucet command can only download hundreds of them. The data format is different. Faucet task reads the moblity_cvx.urdf file while the released large-scale dataset provide only mobility.urdf.

Could you please release the preprocess script or the processed data of other partnet mobility objects?

Rendering with kuafu throws segmentation randomly

Hi,

After I update with the most recent commints on the kuafu renderer, I get segemtantion fault when running the kuafu renderer. Below is a minimal script to hopfully reproduce the issue I face.

import gym
import numpy as np
from mani_skill2.utils.wrappers import RecordEpisode


def main():
    np.set_printoptions(suppress=True, precision=3)
    env = gym.make(
        'TurnFaucet-v0',
        obs_mode=None,
        reward_mode=None,
        control_mode='pd_ee_delta_pose',
        enable_kuafu=True)
    record_dir = './'
    env = RecordEpisode(env, record_dir, render_mode='cameras')

    frames = []
    for i in range(10):
        env.reset()
        while True:
            render_frame = env.render(mode='cameras')
            frames.append(render_frame)

            action = env.action_space.sample()
            obs, reward, done, info = env.step(action)
            if done:
                break


if __name__ == "__main__":
    main()

When I run the above script, I wil l get the following error. Sometimes it will throw a segmentation fault instead after a few trials.

[2022-09-13 00:21:16.995] [kuafu] [info] Camera is not yet usable due to uninitialized context!
[2022-09-13 00:21:16.995] [kuafu] [info] Offscreen mode enabled.
[2022-09-13 00:21:16.995] [kuafu] [warning] Denoiser ON! You must have an NVIDIA GPU with driver version > 470 installed.
2022-09-13 00:21:17,948 - mani_skill2 - WARNING - Only rgb is supported by KuafuRenderer.
trial 0
trial 1
Traceback (most recent call last):
  File "h5py/_objects.pyx", line 201, in h5py._objects.ObjectID.__dealloc__
RuntimeError: Can't decrement id ref count (file write failed: time = Tue Sep 13 00:21:46 2022
, filename = '20220913_002118.h5', file descriptor = 52, errno = 9, error message = 'Bad file descriptor', buf = 0x556c1261aae8, total write size = 64320, bytes this sub-write = 64320, bytes actually written = 18446744073709551615, offset = 18446744073709551615)
Exception ignored in: 'h5py._objects.ObjectID.__dealloc__'
Traceback (most recent call last):
  File "h5py/_objects.pyx", line 201, in h5py._objects.ObjectID.__dealloc__
RuntimeError: Can't decrement id ref count (file write failed: time = Tue Sep 13 00:21:46 2022
, filename = '20220913_002118.h5', file descriptor = 52, errno = 9, error message = 'Bad file descriptor', buf = 0x556c1261aae8, total write size = 64320, bytes this sub-write = 64320, bytes actually written = 18446744073709551615, offset = 18446744073709551615)

Rendering with kuafu of different size gives noisy image

With the kuafu renderer, I tried to change the default camera size from 512 to 1024. This is done by changing this line: https://github.com/haosulab/ManiSkill2/blob/787439b2484190bcbbca2e2daa9aa2cea38816be/mani_skill2/envs/experimental/turn_faucet.py#L74

However, I got noisy rendered results. See the noisy pixels in the sky. Do you know why this might happen?

gravity in pinch_env

The mpm model gravity is (0, 0, -1) in the pinch environment https://github.com/haosulab/ManiSkill2/blob/0f72be61a866752609039b1e797403d10c29e42b/mani_skill2/envs/mpm/pinch_env.py#L59, may I ask why it is not set to -9.8 as in the original MPMModel https://github.com/haosulab/ManiSkill2/blob/0f72be61a866752609039b1e797403d10c29e42b/warp_maniskill/warp/sim/model.py#L357?

Mass for YCB models?

Hello,
I was wondering where mass for YCB models are stored, or are they ignored in simulation?
Thank you!

Install of ManiSkill2 fails

Thanks for open-sourcing the awesome work. I have trouble installing ManiSkill2 on a server and hope you can provide some help. Thanks.

I am using a training platform so I can only run commands inside the docker container. The NVIDIA driver version is 470.57.02. I could not change the version of the NVIDIA driver due to platform limitations. There is no libGLX_nvidia.so file inside the docker. So I download the NVIDIA driver 470.57.02 from the official website and extract all files from the driver file. I find libGLX_nvidia.so.470.57.02 and nvidia_icd.json and save them to /dev/shm/base folder.

I also find the vulkan libs:

The command I run is:

It seems there is something wrong with the render. Could you please give me some hints how to solve this problem?

How to use a more realistic environment?

Glad to see a wonderful simulation tool!
I noticed that the environment shown in the Github teaser (https://github.com/haosulab/ManiSkill2/blob/main/figures/teaser_v2.jpg) looks much more realistic than the environment shown on the website (https://sapien.ucsd.edu/challenges/maniskill/#getting-started). What is the reason for this, and how to use the realistic environment shown in the teaser?

Replay demonstration using the raytracing rendering

Hi,

Thank you for creating this great set of tasks! Are there examples that use the kuafu renderer for replaying the trajectories? Naively replacing the renderer in sapien_env.py results in segmentation fault:

[2022-08-31 14:43:21.660] [kuafu] [warning] Denoiser ON! You must have an NVIDIA GPU with driver version > 470 installed.
[2022-08-31 14:43:22.414] [kuafu] [warning] Mesh not triangulated!
[2022-08-31 14:43:22.414] [kuafu] [warning] Mesh not triangulated!
[2022-08-31 14:43:22.430] [kuafu] [warning] Mesh not triangulated!
[2022-08-31 14:43:22.442] [kuafu] [warning] Mesh not triangulated!
[2022-08-31 14:43:22.442] [kuafu] [warning] Mesh not triangulated!
[2022-08-31 14:43:22.442] [kuafu] [warning] Mesh not triangulated!
Replaying traj_0: : 0step [00:00, ?step/s, control_mode=pd_joint_pos, obs_mode=none][2022-08-31 14:43:22.673] [kuafu] [warning] Mesh not triangulated!
[2022-08-31 14:43:22.673] [kuafu] [warning] Mesh not triangulated!
[2022-08-31 14:43:22.688] [kuafu] [warning] Mesh not triangulated!
[2022-08-31 14:43:22.698] [kuafu] [warning] Mesh not triangulated!
[2022-08-31 14:43:22.698] [kuafu] [warning] Mesh not triangulated!
[2022-08-31 14:43:22.698] [kuafu] [warning] Mesh not triangulated!
/home/xingyu/software/miniconda3/envs/mani_skill2/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
Segmentation fault (core dumped)

Potential issue for soft-body environments due to trimesh

It is reported that trimesh==3.15.8 is not compatible with initial states of our current soft-body environments.
AttributeError: 'ColorVisuals' object has no attribute 'crc'
We need to remove trimesh in initial states or not use pickle.

Setting scales for objects

I was wondering if there is any generic way for setting objects' scale in a scene, e.g. the cube's scale in PickCube-v0, and the object's scale in PickSingleYCB-v0, for domain randomization?

Thanks so much!

Scaling Issues: Difficulty running 64 environments in parallel

Hello Team,

I've been working on a project where I am trying to train reinforcement learning agents using a parallelized setup, similar to the one described in the tutorial.

My machine is equipped with 16 CPU cores, 64GB of memory, and an Nvidia A100 GPU. I have successfully run training using 50 parallel environments, but when I attempt to scale up to 64 environments, I encounter an issue (as attached below).

Despite my machine's resources seeming to be sufficient, this error occurs consistently.

Could you please provide some guidance on how I might successfully scale up to a larger number of environments? Is there a specific resource limitation I am hitting, or could it be due to some software constraint that I might not be aware of?

I appreciate any insights or suggestions you may have!

Best,
Yichao

(mani3) ycliang at node100 in /om2/user/ycliang/gms on main*
$ python train_sb.py
[2023-07-12 18:17:09.759] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:09.759] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:14.224] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:14.224] [svulkan2] [warning] Continue without GLFW.
2023-07-12 18:17:14,919 - mani_skill2 - WARNING - mani_skill2 is not installed with git.
[2023-07-12 18:17:32.910] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:32.910] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:33.344] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:33.344] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:33.390] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:33.390] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:33.419] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:33.419] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:33.429] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:33.430] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:33.452] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:33.452] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:33.602] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:33.602] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:33.772] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:33.772] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:33.778] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:33.778] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:33.784] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:33.784] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:33.804] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:33.804] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:33.835] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:33.835] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:33.885] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:33.885] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:33.902] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:33.902] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:33.929] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:33.929] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:34.042] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:34.042] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:34.081] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:34.081] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:34.084] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:34.084] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:34.116] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:34.116] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:34.180] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:34.180] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:34.241] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:34.241] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:34.349] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:34.349] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:34.487] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:34.487] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:34.548] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:34.548] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:34.591] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:34.591] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:34.641] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:34.641] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:34.761] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:34.761] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:34.801] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:34.801] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:34.805] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:34.805] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:34.906] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:34.906] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:34.921] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:34.921] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:34.923] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:34.923] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:34.962] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:34.962] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:34.986] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:34.986] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:35.031] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:35.031] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:35.108] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:35.108] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:35.207] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:35.207] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:35.217] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:35.217] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:35.233] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:35.233] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:35.239] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:35.239] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:35.240] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:35.240] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:35.267] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:35.267] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:35.317] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:35.317] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:35.345] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:35.345] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:35.353] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:35.353] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:35.387] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:35.387] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:35.402] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:35.402] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:35.404] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:35.404] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:35.474] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:35.475] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:35.476] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:35.476] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:35.499] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:35.499] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:35.519] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:35.519] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:35.547] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:35.547] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:35.600] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:35.600] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:35.606] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:35.606] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:35.611] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:35.611] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:35.635] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:35.636] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:35.653] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:35.653] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:35.700] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:35.700] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:35.726] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:35.726] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:35.726] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:35.726] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:35.726] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:35.726] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:35.766] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:35.766] [svulkan2] [warning] Continue without GLFW.
[2023-07-12 18:17:35.832] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-07-12 18:17:35.832] [svulkan2] [warning] Continue without GLFW.
Process ForkServerProcess-51:
Traceback (most recent call last):
  File "/om2/user/ycliang/miniconda3/envs/mani3/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/om2/user/ycliang/miniconda3/envs/mani3/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/om2/user/ycliang/miniconda3/envs/mani3/lib/python3.11/site-packages/stable_baselines3/common/vec_env/subproc_vec_env.py", line 25, in _worker
    env = env_fn_wrapper.var()
          ^^^^^^^^^^^^^^^^^^^^
  File "/net/vast-storage/scratch/vast/tenenbaum/ycliang/gms/misc.py", line 53, in _init
    env = gym.make(env_id, obs_mode=obs_mode,
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/net/vast-storage/scratch/vast/tenenbaum/ycliang/gym/gym/envs/registration.py", line 235, in make
    return registry.make(id, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/net/vast-storage/scratch/vast/tenenbaum/ycliang/gym/gym/envs/registration.py", line 129, in make
    env = spec.make(**kwargs)
          ^^^^^^^^^^^^^^^^^^^
  File "/net/vast-storage/scratch/vast/tenenbaum/ycliang/gym/gym/envs/registration.py", line 87, in make
    env = self.entry_point(**_kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/om2/user/ycliang/miniconda3/envs/mani3/lib/python3.11/site-packages/mani_skill2/utils/registration.py", line 92, in make
    env = env_spec.make(**kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/om2/user/ycliang/miniconda3/envs/mani3/lib/python3.11/site-packages/mani_skill2/utils/registration.py", line 34, in make
    return self.cls(**_kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/net/vast-storage/scratch/vast/tenenbaum/ycliang/gms/task/pick_cube_rma.py", line 24, in __init__
    super().__init__(*args, **kwargs)
  File "/om2/user/ycliang/miniconda3/envs/mani3/lib/python3.11/site-packages/mani_skill2/envs/pick_and_place/pick_cube.py", line 22, in __init__
    super().__init__(*args, **kwargs)
  File "/om2/user/ycliang/miniconda3/envs/mani3/lib/python3.11/site-packages/mani_skill2/envs/pick_and_place/base_env.py", line 27, in __init__
    super().__init__(*args, **kwargs)
  File "/om2/user/ycliang/miniconda3/envs/mani3/lib/python3.11/site-packages/mani_skill2/envs/sapien_env.py", line 102, in __init__
    self._renderer = sapien.SapienRenderer(**renderer_kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: vk::PhysicalDevice::createDeviceUnique: ErrorInitializationFailed
Traceback (most recent call last):
  File "/net/vast-storage/scratch/vast/tenenbaum/ycliang/gms/train_sb.py", line 201, in <module>
    main()
  File "/net/vast-storage/scratch/vast/tenenbaum/ycliang/gms/train_sb.py", line 115, in main
    env = VecMonitor(env)
          ^^^^^^^^^^^^^^^
  File "/om2/user/ycliang/miniconda3/envs/mani3/lib/python3.11/site-packages/stable_baselines3/common/vec_env/vec_monitor.py", line 39, in __init__
    is_wrapped_with_monitor = venv.env_is_wrapped(Monitor)[0]
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/om2/user/ycliang/miniconda3/envs/mani3/lib/python3.11/site-packages/stable_baselines3/common/vec_env/subproc_vec_env.py", line 186, in env_is_wrapped
    return [remote.recv() for remote in target_remotes]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/om2/user/ycliang/miniconda3/envs/mani3/lib/python3.11/site-packages/stable_baselines3/common/vec_env/subproc_vec_env.py", line 186, in <listcomp>
    return [remote.recv() for remote in target_remotes]
            ^^^^^^^^^^^^^
  File "/om2/user/ycliang/miniconda3/envs/mani3/lib/python3.11/multiprocessing/connection.py", line 249, in recv
    buf = self._recv_bytes()
          ^^^^^^^^^^^^^^^^^^
  File "/om2/user/ycliang/miniconda3/envs/mani3/lib/python3.11/multiprocessing/connection.py", line 413, in _recv_bytes
    buf = self._recv(4)
          ^^^^^^^^^^^^^
  File "/om2/user/ycliang/miniconda3/envs/mani3/lib/python3.11/multiprocessing/connection.py", line 378, in _recv
    chunk = read(handle, remaining)
            ^^^^^^^^^^^^^^^^^^^^^^^
ConnectionResetError: [Errno 104] Connection reset by peer

Converting demonstration to ee does not work

I tried to convert to EE pose by running the following command:
python tools/replay_trajectory.py --traj-path demos/rigid_body_envs/TurnFaucet-v0/5000.h5 --save-traj --target-control-mode pd_ee_delta_pose --obs-mode none

and get the following output

0step [00:00, ?step/s][2022-09-01 12:17:51.855] [SAPIEN] [warning] A second engine will share the same internal structures with the first one. Arguments passed to constructor will be ignored.                                                                                     
[2022-09-01 12:17:51.855] [SAPIEN] [warning] A second renderer will share the same internal context with the first one. Arguments passed to constructor will be ignored.                                                                                                            
[2022-09-01 12:17:51.855] [SAPIEN] [warning] Setting renderer more than once should be avoided.                                                                                                                                                                                     
Episode 0 is not replayed successfully. Skipping                                                                                                                                                                                                                                    
Episode 1 is not replayed successfully. Skipping                                                                                                                                                                                                                                    
Episode 2 is not replayed successfully. Skipping                                                                                                                                                                                                                                    
Episode 3 is not replayed successfully. Skipping                                                                                                                                                                                                                                    
Episode 4 is not replayed successfully. Skipping                     
Episode 5 is not replayed successfully. Skipping                     
Episode 6 is not replayed successfully. Skipping                     
Episode 7 is not replayed successfully. Skipping                     
Episode 8 is not replayed successfully. Skipping                     
Episode 9 is not replayed successfully. Skipping

Is this an issue with conversion of the actions?

ModuleNotFoundError: No module named 'warp' while compiling soft body environments.

Hi. Recently, I faced this error when trying to compile the soft-body environments. I follow the commands in https://haosulab.github.io/ManiSkill2/getting_started/installation.html#warp-maniskill2-version . Do I need to manually install wrap-lang module ? If so , which version should I install?

root@baochen-template-create:/baochen/ManiSkill2# export CUDA_PATH=/usr/local/cuda
root@baochen-template-create:/baochen/ManiSkill2# ${CUDA_PATH}/bin/nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Mon_May__3_19:15:13_PDT_2021
Cuda compilation tools, release 11.3, V11.3.109
Build cuda_11.3.r11.3/compiler.29920130_0
root@baochen-template-create:/baochen/ManiSkill2# gcc --version
gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
root@baochen-template-create:/baochen/ManiSkill2# python3 -m warp_maniskill.build_lib
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/baochen/ManiSkill2/warp_maniskill/build_lib.py", line 12, in <module>
    import warp.config
ModuleNotFoundError: No module named 'warp'

RuntimeError: Cannot find a suitable rendering device

➜ /data/private/ljm/ManiSkill2 python
Python 3.8.5 (default, Sep 4 2020, 07:30:14)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.

import gym
import mani_skill2.envs # import to register all environments in gym

env = gym.make("PickCube-v0", obs_mode="rgbd", control_mode="pd_ee_delta_pose")
{'offscreen_only': True}
Traceback (most recent call last):
File "", line 1, in
File "/opt/conda/lib/python3.8/site-packages/gym/envs/registration.py", line 184, in make
return registry.make(id, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/gym/envs/registration.py", line 106, in make
env = spec.make(kwargs)
File "/opt/conda/lib/python3.8/site-packages/gym/envs/registration.py", line 73, in make
env = self.entry_point(_kwargs)
File "/data/private/ljm/ManiSkill2/mani_skill2/utils/registration.py", line 92, in make
env = env_spec.make(kwargs)
File "/data/private/ljm/ManiSkill2/mani_skill2/utils/registration.py", line 34, in make
return self.cls(_kwargs)
File "/data/private/ljm/ManiSkill2/mani_skill2/envs/pick_and_place/pick_cube.py", line 22, in init
super().init(*args, **kwargs)
File "/data/private/ljm/ManiSkill2/mani_skill2/envs/pick_and_place/base_env.py", line 27, in init
super().init(*args, **kwargs)
File "/data/private/ljm/ManiSkill2/mani_skill2/envs/sapien_env.py", line 106, in init
self._renderer = sapien.SapienRenderer(**renderer_kwargs)
RuntimeError: Cannot find a suitable rendering device

X11 Forwarding?

Hi ManiSkill2,

Thank you guys so much for your work! As someone working with sim RL environments for the first time, I really appreciate all the effort you guys have put into enabling that with ManiSkill2!

At the moment, I've tried using ManiSkill2 from a gpu server but, as you'd expect, there's no X11. As a workaround, I've tried installing something like XQuartz that is said to receive any X11 requests I'd make from ssh, in theory.

Upon using XQuartz and running python examples/demo_manual_control.py -e PickCube-v0, I no longer get the X11 'display environment variable missing' error but am still getting
opts: [] env_kwargs: {} Segmentation fault (core dumped)

Any ideas how I can address this? This is all very new to me and any help would be appreciated! This is all from an M1 macbook for what it's worth. Thanks

Create an actuated single free-floating rigid body

I would like to control the motion of a single free-floating actor (rigid body). As far as I understand it, similar to controlling a revolute joint by torque, now I should control this free-floating object by a wrench (including both a force and a torque vector). Is there an example of adding such free-floating actor with actuation in ManiSkill2/SAPIEN?

My main question is what sapien.Joint should I use in my controller? I was trying to subclass MyPoseController from BaseController. But BaseController would need a list of Joint https://github.com/haosulab/ManiSkill2/blob/5178d73929076e1174d934565a8a11b339aae890/mani_skill2/agents/base_controller.py#L19. Currently Sapien only supports revolute, prismatic and fix joint. How should I handle this free-floating joint with actuation?

Error while running on A100 server. I am trying to run basic demo in A100 server, but having problems due rendering and display.

When I run python -m mani_skill2.examples.demo_random_action command. I get following error.

[2023-02-17 19:03:09.661] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
[2023-02-17 19:03:09.661] [svulkan2] [warning] Continue without GLFW.
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/workspace/sudhiry/rl_ws/ManiSkill2/mani_skill2/examples/demo_random_action.py", line 71, in <module>
    main()
  File "/workspace/sudhiry/rl_ws/ManiSkill2/mani_skill2/examples/demo_random_action.py", line 34, in main
    env: BaseEnv = gym.make(
  File "/opt/conda/lib/python3.8/site-packages/gym/envs/registration.py", line 235, in make
    return registry.make(id, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/gym/envs/registration.py", line 129, in make
    env = spec.make(**kwargs)
  File "/opt/conda/lib/python3.8/site-packages/gym/envs/registration.py", line 87, in make
    env = self.entry_point(**_kwargs)
  File "/workspace/sudhiry/rl_ws/ManiSkill2/mani_skill2/utils/registration.py", line 92, in make
    env = env_spec.make(**kwargs)
  File "/workspace/sudhiry/rl_ws/ManiSkill2/mani_skill2/utils/registration.py", line 34, in make
    return self.cls(**_kwargs)
  File "/workspace/sudhiry/rl_ws/ManiSkill2/mani_skill2/envs/pick_and_place/pick_cube.py", line 22, in __init__
    super().__init__(*args, **kwargs)
  File "/workspace/sudhiry/rl_ws/ManiSkill2/mani_skill2/envs/pick_and_place/base_env.py", line 27, in __init__
    super().__init__(*args, **kwargs)
  File "/workspace/sudhiry/rl_ws/ManiSkill2/mani_skill2/envs/sapien_env.py", line 110, in __init__
    self._renderer = sapien.SapienRenderer(**renderer_kwargs)
RuntimeError: Cannot find a suitable rendering device

How to run this in a headless mode, where redering is not required?

Error when runing sb3_ppo_liftcube_rgbd

Hi,

I run the example rl code

python examples/tutorials/reinforcement-learning/sb3_ppo_liftcube_rgbd.py

but get the following error:

Traceback (most recent call last):
File "/home/user/Projects/manipulation/mani_env/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/home/user/Projects/manipulation/mani_env/lib/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/user/Projects/manipulation/stable-baselines3/stable_baselines3/common/vec_env/subproc_vec_env.py", line 30, in _worker
env = _patch_env(env_fn_wrapper.var())
^^^^^^^^^^^^^^^^^^^^
File "/home/user/Projects/manipulation/ManiSkill2/examples/tutorials/reinforcement-learning/sb3_ppo_liftcube_rgbd.py", line 293, in make_env
env = RecordEpisode(
^^^^^^^^^^^^^^
File "/home/user/Projects/manipulation/ManiSkill2/mani_skill2/utils/wrappers/record.py", line 136, in init
self.render_mode = render_mode
^^^^^^^^^^^^^^^^
AttributeError: property 'render_mode' of 'RecordEpisode' object has no setter
Traceback (most recent call last):
File "/home/user/Projects/manipulation/ManiSkill2/examples/tutorials/reinforcement-learning/sb3_ppo_liftcube_rgbd.py", line 405, in
main()
File "/home/user/Projects/manipulation/ManiSkill2/examples/tutorials/reinforcement-learning/sb3_ppo_liftcube_rgbd.py", line 312, in main
eval_env = SubprocVecEnv([env_fn for _ in range(1)])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/Projects/manipulation/stable-baselines3/stable_baselines3/common/vec_env/subproc_vec_env.py", line 121, in init
observation_space, action_space = self.remotes[0].recv()
^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/Projects/manipulation/mani_env/lib/python3.11/multiprocessing/connection.py", line 249, in recv
buf = self._recv_bytes()
^^^^^^^^^^^^^^^^^^
File "/home/user/Projects/manipulation/mani_env/lib/python3.11/multiprocessing/connection.py", line 413, in _recv_bytes
buf = self._recv(4)
^^^^^^^^^^^^^
File "/home/user/Projects/manipulation/mani_env/lib/python3.11/multiprocessing/connection.py", line 378, in _recv
chunk = read(handle, remaining)

==================

(My environment is newly set up)

could you tell me where the PD controller code ?

I hope to deploy the agent trained by maniskill2 to a real machine .is there any documentation or code to learn ?

Error while training a DAPG agent under "env_name=PegInsertionSide-v0"

Hi, when I trained a DAPG agent under "env_name=PegInsertionSide-v0" using the script:

DISPLAY="" python maniskill2_learn/apis/run_rl.py configs/mfrl/dapg/maniskill2_pn.py 
--work-dir /data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-main/Result/PegInsertionSide-v0/DAPG 
--gpu-ids 2 --cfg-options "env_cfg.env_name=PegInsertionSide-v0" "env_cfg.obs_mode=pointcloud" "env_cfg.n_points=1200" "env_cfg.reward_mode=dense" "rollout_cfg.num_procs=5"
 "env_cfg.control_mode=pd_ee_delta_pose" "env_cfg.obs_frame=ee" "agent_cfg.demo_replay_cfg.capacity=20000" 
"agent_cfg.demo_replay_cfg.cache_size=20000" "agent_cfg.demo_replay_cfg.dynamic_loading=True" "agent_cfg.demo_replay_cfg.num_samples=-1" 
"agent_cfg.demo_replay_cfg.buffer_filenames=/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-main-old/demos/rigid_body/PegInsertionSide-v0/trajectory.h5"  
"eval_cfg.num=100" "eval_cfg.save_traj=False" "eval_cfg.save_video=True"

the error came out like:

PegInsertionSide-v0-train - (collect_env.py:133) - INFO - 2023-04-22,09:52:31 - meta_collect_time: 2023-04-22-02:52:31
PegInsertionSide-v0-train - (collect_env.py:133) - INFO - 2023-04-22,09:52:31 - PYRL: version: 1.8.0b0
PegInsertionSide-v0-train - (collect_env.py:133) - INFO - 2023-04-22,09:52:31 - ManiSkill2: 
PegInsertionSide-v0-train - (run_rl.py:243) - INFO - 2023-04-22,09:52:31 - Initialize torch!
PegInsertionSide-v0-train - (run_rl.py:245) - INFO - 2023-04-22,09:52:31 - Finish Initialize torch!
PegInsertionSide-v0-train - (run_rl.py:253) - INFO - 2023-04-22,09:52:31 - Build agent!
PegInsertionSide-v0-train - (replay_buffer.py:59) - INFO - 2023-04-22,09:52:31 - Load 1 files!
PegInsertionSide-v0-train - (logger.py:155) - INFO - 2023-04-22,09:52:31 - 0%|          | 0/1 [00:00<?, ?it/s]
PegInsertionSide-v0-train - (logger.py:155) - INFO - 2023-04-22,09:52:31 - 100%|##########| 1/1 [00:00<00:00,  5.68it/s]
PegInsertionSide-v0-train - (replay_buffer.py:62) - INFO - 2023-04-22,09:52:31 - Load 1 files with 154419 samples in total!
Traceback (most recent call last):
  File "maniskill2_learn/apis/run_rl.py", line 487, in <module>
    main()
  File "maniskill2_learn/apis/run_rl.py", line 452, in main
    run_one_process(0, 1, args, cfg)
  File "maniskill2_learn/apis/run_rl.py", line 432, in run_one_process
    main_rl(rollout, evaluator, replay, args, cfg, expert_replay=expert_replay, recent_traj_replay=recent_traj_replay)
  File "maniskill2_learn/apis/run_rl.py", line 254, in main_rl
    agent = build_agent(cfg.agent_cfg)
  File "/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-new/maniskill2_learn/methods/builder.py", line 12, in build_agent
    return build_from_cfg(cfg, agent_type, default_args)
  File "/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-new/maniskill2_learn/utils/meta/registry.py", line 136, in build_from_cfg
    return obj_cls(**args)
  File "/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-new/maniskill2_learn/methods/mfrl/ppo.py", line 134, in __init__
    self.demo_replay = build_replay(demo_replay_cfg)
  File "/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-new/maniskill2_learn/env/builder.py", line 27, in build_replay
    return build_from_cfg(cfg, REPLAYS, default_args)
  File "/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-new/maniskill2_learn/utils/meta/registry.py", line 136, in build_from_cfg
    return obj_cls(**args)
  File "/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-new/maniskill2_learn/env/replay_buffer.py", line 77, in __init__
    self.file_loader = FileCache(
  File "/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-new/maniskill2_learn/utils/file/cache_utils.py", line 450, in __init__
    self.shared_buffer = create_shared_dict_array_from_files(filenames, capacity, data_coder, keys, keys_map=keys_map)
  File "/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-new/maniskill2_learn/utils/file/cache_utils.py", line 138, in create_shared_dict_array_from_files
    item = DictArray(item)
  File "/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-new/maniskill2_learn/utils/data/dict_array.py", line 763, in __init__
    self.assert_shape(self.memory, self.capacity)
  File "/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-new/maniskill2_learn/utils/data/dict_array.py", line 802, in assert_shape
    assert cls.check_shape(memory, capacity), f"The first dimension is not {capacity}!"
AssertionError: The first dimension is not 137!
Exception ignored in: <function ReplayMemory.__del__ at 0x7fab7d260790>
Traceback (most recent call last):
  File "/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-new/maniskill2_learn/env/replay_buffer.py", line 257, in __del__
    self.close()
  File "/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-new/maniskill2_learn/env/replay_buffer.py", line 253, in close
    if self.file_loader is not None:
  File "/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-new/maniskill2_learn/env/replay_buffer.py", line 154, in __getattr__
    return getattr(self.memory, key, None)
  File "/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-new/maniskill2_learn/env/replay_buffer.py", line 154, in __getattr__
    return getattr(self.memory, key, None)
  File "/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-new/maniskill2_learn/env/replay_buffer.py", line 154, in __getattr__
    return getattr(self.memory, key, None)
  [Previous line repeated 995 more times]
RecursionError: maximum recursion depth exceeded
Exception ignored in: <function SharedGDict.__del__ at 0x7fac5a2bbd30>
Traceback (most recent call last):
  File "/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-new/maniskill2_learn/utils/data/dict_array.py", line 928, in __del__
  File "/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-new/maniskill2_learn/utils/data/dict_array.py", line 913, in _unlink
  File "/data/home-gxu/lxt21/.conda/envs/sapien/lib/python3.8/multiprocessing/shared_memory.py", line 237, in unlink
ImportError: sys.meta_path is None, Python is likely shutting down
/data/home-gxu/lxt21/.conda/envs/sapien/lib/python3.8/multiprocessing/resource_tracker.py:203: UserWarning: resource_tracker: There appear to be 18 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d

How can I solve it?

Do you have exhaustive results on your benchmark tasks?

Hi, ManiSkill2 is such a wonderful work and I am very interested in it! I'd like to know if there are any exhaustive results on all of your tasks in this benchmark, with every possible IL/RL algorithm and input format. In your paper, I can only find the results of:

BC
PPO+DAPG

But obviously, there are many other mainstream algos to be tested. In fact, the most thing I want to know are results with pure RL algorithms with point-cloud-based input on all of your tasks. It would be great if you could provide these results. Thanks!

Google Colab failed "No module named 'mani_skill2'"

Hello!

It looks like Google Colab probably updates again...now the second tutorial on RL fails

https://colab.research.google.com/github/haosulab/ManiSkill2/blob/main/examples/tutorials/2_reinforcement_learning.ipynb

This import fails import mani_skill2.envs

ModuleNotFoundError                       Traceback (most recent call last)
[<ipython-input-3-4c7fdd177ecb>](https://localhost:8080/#) in <cell line: 6>()
      4 from tqdm.notebook import tqdm
      5 import numpy as np
----> 6 import mani_skill2.envs
      7 import matplotlib.pyplot as plt
      8 import torch.nn as nn

ModuleNotFoundError: No module named 'mani_skill2'

the pip installation log is below

Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Collecting setuptools==65.5.0
  Downloading setuptools-65.5.0-py3-none-any.whl (1.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 15.2 MB/s eta 0:00:00
Installing collected packages: setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 67.7.2
    Uninstalling setuptools-67.7.2:
      Successfully uninstalled setuptools-67.7.2
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
ipython 7.34.0 requires jedi>=0.16, which is not installed.
cvxpy 1.3.1 requires setuptools>65.5.1, but you have setuptools 65.5.0 which is incompatible.
Successfully installed setuptools-65.5.0
Reading package lists... Done
Building dependency tree       
Reading state information... Done
libvulkan-dev is already the newest version (1.2.131.2-1).
libvulkan-dev set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 24 not upgraded.
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Collecting mani_skill2
  Downloading mani_skill2-0.4.2-py3-none-any.whl (12.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.4/12.4 MB 60.4 MB/s eta 0:00:00
Collecting stable_baselines3
  Downloading stable_baselines3-1.8.0-py3-none-any.whl (174 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 174.5/174.5 kB 20.3 MB/s eta 0:00:00
Requirement already satisfied: numpy<1.24 in /usr/local/lib/python3.9/dist-packages (from mani_skill2) (1.22.4)
Requirement already satisfied: scipy in /usr/local/lib/python3.9/dist-packages (from mani_skill2) (1.10.1)
Collecting sapien==2.2.1
  Downloading sapien-2.2.1-cp39-cp39-manylinux2014_x86_64.whl (39.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 39.1/39.1 MB 14.7 MB/s eta 0:00:00
Requirement already satisfied: tqdm in /usr/local/lib/python3.9/dist-packages (from mani_skill2) (4.65.0)
Requirement already satisfied: tabulate in /usr/local/lib/python3.9/dist-packages (from mani_skill2) (0.8.10)
Collecting trimesh
  Downloading trimesh-3.21.5-py3-none-any.whl (680 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 680.8/680.8 kB 61.5 MB/s eta 0:00:00
Collecting gym<=0.21.0,>=0.18.3
  Downloading gym-0.21.0.tar.gz (1.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 76.3 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Requirement already satisfied: h5py in /usr/local/lib/python3.9/dist-packages (from mani_skill2) (3.8.0)
Requirement already satisfied: imageio[ffmpeg] in /usr/local/lib/python3.9/dist-packages (from mani_skill2) (2.25.1)
Collecting GitPython
  Downloading GitPython-3.1.31-py3-none-any.whl (184 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 184.3/184.3 kB 24.6 MB/s eta 0:00:00
Collecting transforms3d
  Downloading transforms3d-0.4.1.tar.gz (1.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.4/1.4 MB 85.0 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Requirement already satisfied: opencv-python in /usr/local/lib/python3.9/dist-packages (from mani_skill2) (4.7.0.72)
Requirement already satisfied: pyyaml in /usr/local/lib/python3.9/dist-packages (from mani_skill2) (6.0)
Collecting rtree
  Downloading Rtree-1.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 73.4 MB/s eta 0:00:00
Requirement already satisfied: gdown>=4.6.0 in /usr/local/lib/python3.9/dist-packages (from mani_skill2) (4.6.6)
Requirement already satisfied: requests>=2.22 in /usr/local/lib/python3.9/dist-packages (from sapien==2.2.1->mani_skill2) (2.27.1)
Requirement already satisfied: cloudpickle in /usr/local/lib/python3.9/dist-packages (from stable_baselines3) (2.2.1)
Requirement already satisfied: torch>=1.11 in /usr/local/lib/python3.9/dist-packages (from stable_baselines3) (2.0.0+cu118)
Requirement already satisfied: pandas in /usr/local/lib/python3.9/dist-packages (from stable_baselines3) (1.5.3)
Collecting importlib-metadata~=4.13
  Downloading importlib_metadata-4.13.0-py3-none-any.whl (23 kB)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.9/dist-packages (from stable_baselines3) (3.7.1)
Requirement already satisfied: six in /usr/local/lib/python3.9/dist-packages (from gdown>=4.6.0->mani_skill2) (1.16.0)
Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.9/dist-packages (from gdown>=4.6.0->mani_skill2) (4.11.2)
Requirement already satisfied: filelock in /usr/local/lib/python3.9/dist-packages (from gdown>=4.6.0->mani_skill2) (3.11.0)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.9/dist-packages (from importlib-metadata~=4.13->stable_baselines3) (3.15.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.9/dist-packages (from torch>=1.11->stable_baselines3) (3.1.2)
Requirement already satisfied: triton==2.0.0 in /usr/local/lib/python3.9/dist-packages (from torch>=1.11->stable_baselines3) (2.0.0)
Requirement already satisfied: sympy in /usr/local/lib/python3.9/dist-packages (from torch>=1.11->stable_baselines3) (1.11.1)
Requirement already satisfied: networkx in /usr/local/lib/python3.9/dist-packages (from torch>=1.11->stable_baselines3) (3.1)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.9/dist-packages (from torch>=1.11->stable_baselines3) (4.5.0)
Requirement already satisfied: lit in /usr/local/lib/python3.9/dist-packages (from triton==2.0.0->torch>=1.11->stable_baselines3) (16.0.1)
Requirement already satisfied: cmake in /usr/local/lib/python3.9/dist-packages (from triton==2.0.0->torch>=1.11->stable_baselines3) (3.25.2)
Collecting gitdb<5,>=4.0.1
  Downloading gitdb-4.0.10-py3-none-any.whl (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.7/62.7 kB 8.6 MB/s eta 0:00:00
Requirement already satisfied: pillow>=8.3.2 in /usr/local/lib/python3.9/dist-packages (from imageio[ffmpeg]->mani_skill2) (8.4.0)
Requirement already satisfied: psutil in /usr/local/lib/python3.9/dist-packages (from imageio[ffmpeg]->mani_skill2) (5.9.5)
Requirement already satisfied: imageio-ffmpeg in /usr/local/lib/python3.9/dist-packages (from imageio[ffmpeg]->mani_skill2) (0.4.8)
Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.9/dist-packages (from matplotlib->stable_baselines3) (4.39.3)
Requirement already satisfied: importlib-resources>=3.2.0 in /usr/local/lib/python3.9/dist-packages (from matplotlib->stable_baselines3) (5.12.0)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.9/dist-packages (from matplotlib->stable_baselines3) (23.1)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib->stable_baselines3) (1.4.4)
Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.9/dist-packages (from matplotlib->stable_baselines3) (2.8.2)
Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib->stable_baselines3) (1.0.7)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.9/dist-packages (from matplotlib->stable_baselines3) (0.11.0)
Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib->stable_baselines3) (3.0.9)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.9/dist-packages (from pandas->stable_baselines3) (2022.7.1)
Collecting smmap<6,>=3.0.1
  Downloading smmap-5.0.0-py3-none-any.whl (24 kB)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.9/dist-packages (from requests>=2.22->sapien==2.2.1->mani_skill2) (1.26.15)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/dist-packages (from requests>=2.22->sapien==2.2.1->mani_skill2) (2022.12.7)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.9/dist-packages (from requests>=2.22->sapien==2.2.1->mani_skill2) (3.4)
Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.9/dist-packages (from requests>=2.22->sapien==2.2.1->mani_skill2) (2.0.12)
Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.9/dist-packages (from beautifulsoup4->gdown>=4.6.0->mani_skill2) (2.4.1)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.9/dist-packages (from jinja2->torch>=1.11->stable_baselines3) (2.1.2)
Requirement already satisfied: PySocks!=1.5.7,>=1.5.6 in /usr/local/lib/python3.9/dist-packages (from requests>=2.22->sapien==2.2.1->mani_skill2) (1.7.1)
Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.9/dist-packages (from sympy->torch>=1.11->stable_baselines3) (1.3.0)
Building wheels for collected packages: gym, transforms3d
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  Building wheel for gym (setup.py) ... error
  ERROR: Failed building wheel for gym
  Running setup.py clean for gym
  Building wheel for transforms3d (setup.py) ... done
  Created wheel for transforms3d: filename=transforms3d-0.4.1-py3-none-any.whl size=1376772 sha256=f226294a85ba5b031f5eafc5996a6f2980d3853053faf7090fe8b84e42d07a65
  Stored in directory: /root/.cache/pip/wheels/a5/2d/e4/5d7de7a69ab4e6301b3125441725d47fe25b9e60e7d670ad21
Successfully built transforms3d
Failed to build gym
Installing collected packages: trimesh, transforms3d, smmap, rtree, importlib-metadata, gym, sapien, gitdb, GitPython, mani_skill2, stable_baselines3
  Attempting uninstall: importlib-metadata
    Found existing installation: importlib-metadata 6.4.1
    Uninstalling importlib-metadata-6.4.1:
      Successfully uninstalled importlib-metadata-6.4.1
  Attempting uninstall: gym
    Found existing installation: gym 0.25.2
    Uninstalling gym-0.25.2:
      Successfully uninstalled gym-0.25.2
  Running setup.py install for gym ... done
  DEPRECATION: gym was installed using the legacy 'setup.py install' method, because a wheel could not be built for it. pip 23.1 will enforce this behaviour change. A possible replacement is to fix the wheel build issue reported above. Discussion can be found at https://github.com/pypa/pip/issues/8368
Successfully installed GitPython-3.1.31 gitdb-4.0.10 gym-0.21.0 importlib-metadata-4.13.0 mani_skill2-0.4.2 rtree-1.0.1 sapien-2.2.1 smmap-5.0.0 stable_baselines3-1.8.0 transforms3d-0.4.1 trimesh-3.21.5
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Requirement already satisfied: gdown in /usr/local/lib/python3.9/dist-packages (4.6.6)
Collecting gdown
  Downloading gdown-4.7.1-py3-none-any.whl (15 kB)
Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.9/dist-packages (from gdown) (4.11.2)
Requirement already satisfied: tqdm in /usr/local/lib/python3.9/dist-packages (from gdown) (4.65.0)
Requirement already satisfied: filelock in /usr/local/lib/python3.9/dist-packages (from gdown) (3.11.0)
Requirement already satisfied: six in /usr/local/lib/python3.9/dist-packages (from gdown) (1.16.0)
Requirement already satisfied: requests[socks] in /usr/local/lib/python3.9/dist-packages (from gdown) (2.27.1)
Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.9/dist-packages (from beautifulsoup4->gdown) (2.4.1)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.9/dist-packages (from requests[socks]->gdown) (1.26.15)
Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.9/dist-packages (from requests[socks]->gdown) (2.0.12)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.9/dist-packages (from requests[socks]->gdown) (3.4)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/dist-packages (from requests[socks]->gdown) (2022.12.7)
Requirement already satisfied: PySocks!=1.5.7,>=1.5.6 in /usr/local/lib/python3.9/dist-packages (from requests[socks]->gdown) (1.7.1)
Installing collected packages: gdown
  Attempting uninstall: gdown
    Found existing installation: gdown 4.6.6
    Uninstalling gdown-4.6.6:
      Successfully uninstalled gdown-4.6.6
Successfully installed gdown-4.7.1
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Collecting sapien==2.0.0.dev20230405
  Downloading https://storage1.ucsd.edu/wheels/sapien-dev/sapien-2.0.0.dev20230405-cp39-cp39-manylinux2014_x86_64.whl (39.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 39.1/39.1 MB 13.6 MB/s eta 0:00:00
Requirement already satisfied: transforms3d>=0.3 in /usr/local/lib/python3.9/dist-packages (from sapien==2.0.0.dev20230405) (0.4.1)
Requirement already satisfied: opencv-python>=4.0 in /usr/local/lib/python3.9/dist-packages (from sapien==2.0.0.dev20230405) (4.7.0.72)
Requirement already satisfied: requests>=2.22 in /usr/local/lib/python3.9/dist-packages (from sapien==2.0.0.dev20230405) (2.27.1)
Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.9/dist-packages (from sapien==2.0.0.dev20230405) (1.22.4)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.9/dist-packages (from requests>=2.22->sapien==2.0.0.dev20230405) (1.26.15)
Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.9/dist-packages (from requests>=2.22->sapien==2.0.0.dev20230405) (2.0.12)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.9/dist-packages (from requests>=2.22->sapien==2.0.0.dev20230405) (3.4)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/dist-packages (from requests>=2.22->sapien==2.0.0.dev20230405) (2022.12.7)
Installing collected packages: sapien
  Attempting uninstall: sapien
    Found existing installation: sapien 2.2.1
    Uninstalling sapien-2.2.1:
      Successfully uninstalled sapien-2.2.1
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
mani-skill2 0.4.2 requires sapien==2.2.1, but you have sapien 2.0.0.dev20230405 which is incompatible.
Successfully installed sapien-2.0.0.dev20230405

Questions about mpm state in soft body envs

Hello, I notice that you use mpm state to record particle state in soft body envs, but I don't know the exact representation of mpm state. I want to know what 'x', 'v', 'F', 'C' and 'vol' in mpm state refer to.

Google Colab Your session crashed for an unknown reason

Hello! I'm running into an issue when trying to run the provided 1_quickstart.ipynb on Google Colab (https://colab.research.google.com/github/haosulab/ManiSkill2/blob/main/examples/tutorials/1_quickstart.ipynb). My runtime has GPU. Is anyone encountering this as well?

Here's the log

Timestamp	Level	Message
Apr 24, 2023, 9:58:50 AM	WARNING	WARNING:root:kernel 015636f9-1898-49cf-9191-a9efcd9d7ae8 restarted
Apr 24, 2023, 9:58:50 AM	INFO	KernelRestarter: restarting kernel (1/5), keep random ports
Apr 24, 2023, 9:58:48 AM	WARNING	[2023-04-24 13:58:48.303] [svulkan2] [error] Some required Vulkan extension is not present. You may not use the renderer to render, however, CPU resources will be still available.
Apr 24, 2023, 9:58:48 AM	WARNING	[2023-04-24 13:58:48.298] [svulkan2] [warning] Continue without GLFW.
Apr 24, 2023, 9:58:48 AM	WARNING	[2023-04-24 13:58:48.298] [svulkan2] [error] GLFW error: X11: The DISPLAY environment variable is missing
Apr 24, 2023, 9:48:47 AM	INFO	Kernel started: 015636f9-1898-49cf-9191-a9efcd9d7ae8, name: python3
Apr 24, 2023, 9:48:34 AM	INFO	Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
Apr 24, 2023, 9:48:34 AM	INFO	http://172\.28\.0\.12:9000/
Apr 24, 2023, 9:48:34 AM	INFO	Jupyter Notebook 6.4.8 is running at:
Apr 24, 2023, 9:48:34 AM	INFO	Serving notebooks from local directory: /
Apr 24, 2023, 9:48:33 AM	INFO	Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
Apr 24, 2023, 9:48:33 AM	INFO	http://172\.28\.0\.2:9000/
Apr 24, 2023, 9:48:33 AM	INFO	Jupyter Notebook 6.3.0 is running at:
Apr 24, 2023, 9:48:33 AM	INFO	Serving notebooks from local directory: /
Apr 24, 2023, 9:48:33 AM	WARNING	ModuleNotFoundError: No module named 'jupyter_server'
Apr 24, 2023, 9:48:33 AM	WARNING	from jupyter_server.base.handlers import JupyterHandler
Apr 24, 2023, 9:48:33 AM	WARNING	File "/usr/local/lib/python3.9/dist-packages/panel/io/jupyter_server_extension.py", line 53, in <module>
Apr 24, 2023, 9:48:33 AM	WARNING	File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
Apr 24, 2023, 9:48:33 AM	WARNING	File "<frozen importlib._bootstrap_external>", line 850, in exec_module
Apr 24, 2023, 9:48:33 AM	WARNING	File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
Apr 24, 2023, 9:48:33 AM	WARNING	File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
Apr 24, 2023, 9:48:33 AM	WARNING	File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
Apr 24, 2023, 9:48:33 AM	WARNING	File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
Apr 24, 2023, 9:48:33 AM	WARNING	return _bootstrap._gcd_import(name[level:], package, level)
Apr 24, 2023, 9:48:33 AM	WARNING	File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
Apr 24, 2023, 9:48:33 AM	WARNING	mod = importlib.import_module(modulename)
Apr 24, 2023, 9:48:33 AM	WARNING	File "/usr/local/lib/python3.9/dist-packages/notebook/notebookapp.py", line 2033, in init_server_extensions
Apr 24, 2023, 9:48:33 AM	WARNING	Traceback (most recent call last):
Apr 24, 2023, 9:48:33 AM	WARNING	Error loading server extension panel.io.jupyter_server_extension
Apr 24, 2023, 9:48:29 AM	INFO	google.colab serverextension initialized.
Apr 24, 2023, 9:48:29 AM	INFO	google.colab serverextension initialized.
Apr 24, 2023, 9:48:29 AM	INFO	Authentication of /metrics is OFF, since other authentication is disabled.
Apr 24, 2023, 9:48:29 AM	INFO	Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret
Apr 24, 2023, 9:48:29 AM	WARNING	/root/.jupyter/jupyter_notebook_config.json
Apr 24, 2023, 9:48:29 AM	WARNING	/root/.local/etc/jupyter/jupyter_notebook_config.json
Apr 24, 2023, 9:48:29 AM	WARNING	/usr/etc/jupyter/jupyter_notebook_config.json
Apr 24, 2023, 9:48:29 AM	WARNING	/usr/local/etc/jupyter/jupyter_notebook_config.json
Apr 24, 2023, 9:48:29 AM	WARNING	/usr/local/etc/jupyter/jupyter_notebook_config.d/panel-client-jupyter.json
Apr 24, 2023, 9:48:29 AM	WARNING	/etc/jupyter/jupyter_notebook_config.json
Apr 24, 2023, 9:48:29 AM	INFO	Authentication of /metrics is OFF, since other authentication is disabled.
Apr 24, 2023, 9:48:29 AM	INFO	Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret
Apr 24, 2023, 9:48:29 AM	WARNING	/root/.jupyter/jupyter_notebook_config.json
Apr 24, 2023, 9:48:29 AM	WARNING	/root/.local/etc/jupyter/jupyter_notebook_config.json
Apr 24, 2023, 9:48:29 AM	WARNING	/usr/etc/jupyter/jupyter_notebook_config.json
Apr 24, 2023, 9:48:29 AM	WARNING	/usr/local/etc/jupyter/jupyter_notebook_config.json
Apr 24, 2023, 9:48:29 AM	WARNING	/usr/local/etc/jupyter/jupyter_notebook_config.d/panel-client-jupyter.json
Apr 24, 2023, 9:48:29 AM	WARNING	/etc/jupyter/jupyter_notebook_config.json
Apr 24, 2023, 9:48:28 AM	WARNING	[D 13:48:28.825 NotebookApp] Loaded config file: /root/.jupyter/jupyter_notebook_config.py
Apr 24, 2023, 9:48:28 AM	WARNING	[D 13:48:28.818 NotebookApp] Looking for jupyter_notebook_config in /root/.jupyter
Apr 24, 2023, 9:48:28 AM	WARNING	[D 13:48:28.816 NotebookApp] Looking for jupyter_notebook_config in /root/.local/etc/jupyter
Apr 24, 2023, 9:48:28 AM	WARNING	[D 13:48:28.816 NotebookApp] Looking for jupyter_notebook_config in /usr/etc/jupyter
Apr 24, 2023, 9:48:28 AM	WARNING	[D 13:48:28.815 NotebookApp] Loaded config file: /usr/local/etc/jupyter/jupyter_notebook_config.json
Apr 24, 2023, 9:48:28 AM	WARNING	[D 13:48:28.812 NotebookApp] Looking for jupyter_notebook_config in /usr/local/etc/jupyter
Apr 24, 2023, 9:48:28 AM	WARNING	[D 13:48:28.812 NotebookApp] Loaded config file: /etc/jupyter/jupyter_notebook_config.py
Apr 24, 2023, 9:48:28 AM	WARNING	[D 13:48:28.808 NotebookApp] Looking for jupyter_notebook_config in /etc/jupyter
Apr 24, 2023, 9:48:28 AM	WARNING	[D 13:48:28.806 NotebookApp] Looking for jupyter_config in /root/.jupyter
Apr 24, 2023, 9:48:28 AM	WARNING	[D 13:48:28.795 NotebookApp] Looking for jupyter_config in /root/.local/etc/jupyter
Apr 24, 2023, 9:48:28 AM	WARNING	[D 13:48:28.795 NotebookApp] Looking for jupyter_config in /usr/etc/jupyter
Apr 24, 2023, 9:48:28 AM	WARNING	[D 13:48:28.777 NotebookApp] Looking for jupyter_config in /usr/local/etc/jupyter
Apr 24, 2023, 9:48:28 AM	WARNING	[D 13:48:28.776 NotebookApp] Looking for jupyter_config in /etc/jupyter
Apr 24, 2023, 9:48:28 AM	WARNING	[D 13:48:28.769 NotebookApp] Searching ['/root/.jupyter', '/root/.local/etc/jupyter', '/usr/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
Apr 24, 2023, 9:48:28 AM	WARNING	[D 13:48:28.769 NotebookApp] Loaded config file: /root/.jupyter/jupyter_notebook_config.py
Apr 24, 2023, 9:48:28 AM	WARNING	[D 13:48:28.765 NotebookApp] Looking for jupyter_notebook_config in /root/.jupyter
Apr 24, 2023, 9:48:28 AM	WARNING	[D 13:48:28.764 NotebookApp] Looking for jupyter_notebook_config in /root/.local/etc/jupyter
Apr 24, 2023, 9:48:28 AM	WARNING	[D 13:48:28.764 NotebookApp] Looking for jupyter_notebook_config in /usr/etc/jupyter
Apr 24, 2023, 9:48:28 AM	WARNING	[D 13:48:28.764 NotebookApp] Loaded config file: /usr/local/etc/jupyter/jupyter_notebook_config.json
Apr 24, 2023, 9:48:28 AM	WARNING	[D 13:48:28.762 NotebookApp] Looking for jupyter_notebook_config in /usr/local/etc/jupyter
Apr 24, 2023, 9:48:28 AM	WARNING	[D 13:48:28.761 NotebookApp] Loaded config file: /etc/jupyter/jupyter_notebook_config.py
Apr 24, 2023, 9:48:28 AM	WARNING	[D 13:48:28.759 NotebookApp] Looking for jupyter_notebook_config in /etc/jupyter
Apr 24, 2023, 9:48:28 AM	WARNING	[D 13:48:28.758 NotebookApp] Looking for jupyter_config in /root/.jupyter
Apr 24, 2023, 9:48:28 AM	WARNING	[D 13:48:28.758 NotebookApp] Looking for jupyter_config in /root/.local/etc/jupyter
Apr 24, 2023, 9:48:28 AM	WARNING	[D 13:48:28.757 NotebookApp] Looking for jupyter_config in /usr/etc/jupyter
Apr 24, 2023, 9:48:28 AM	WARNING	[D 13:48:28.742 NotebookApp] Looking for jupyter_config in /usr/local/etc/jupyter
Apr 24, 2023, 9:48:28 AM	WARNING	[D 13:48:28.738 NotebookApp] Looking for jupyter_config in /etc/jupyter
Apr 24, 2023, 9:48:28 AM	WARNING	[D 13:48:28.738 NotebookApp] Searching ['/root/.jupyter', '/root/.local/etc/jupyter', '/usr/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files

Error from replaying trajectory of "PlugCharger-v0"

Hi,

I am training a model under "env=PlugCharger-v0", I wanted to convert the control mode from pd_joint_pos to pd_ee_delta_pos by running the replay_trajectory file, but it was unsuccessful, and it's the same story for replaying "PegInsertionSide-v0". Does it mean it's unchangeable? If I want to use end-effector control, do I have to manually compute the results from joint control data?

Is there any plan to support cloth-like object manipulation environments in the future?

Hey, there.
Glad to see a wonderful simulation tool for versatile manipulation tasks. I saw soft-body is already supported in the environments, and I am trying to customise an environment for other cloth-like objects manipulation task, so I was wondering if current Maniskill2 has such features and would you like to give any hints about that? Thanks.

Error while training under "env_name=MoveBucket-v1"

Hi! When I was training under "env_name=MoveBucket-v1" like:

python maniskill2_learn/apis/run_rl.py configs/mfrl/ppo/maniskill2_pn.py 
--work-dir /data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-main/Result/MoveBucket-v1/PPO 
--gpu-ids 0 --cfg-options "env_cfg.env_name=MoveBucket-v1" 
"env_cfg.obs_mode=pointcloud" "env_cfg.n_points=1200" "env_cfg.control_mode=pd_joint_delta_pos" "env_cfg.reward_mode=dense" 
"rollout_cfg.num_procs=5" "eval_cfg.num=100" "eval_cfg.save_traj=False" "eval_cfg.save_video=True" "eval_cfg.num_procs=5"

there was an error:

MoveBucket-v1-train - (run_rl.py:347) - INFO - 2023-04-10,03:46:16 - Set random seed to 144182442
MoveBucket-v1-train - (run_rl.py:351) - INFO - 2023-04-10,03:46:16 - Build replay buffer!
MoveBucket-v1-train - (run_rl.py:370) - INFO - 2023-04-10,03:46:16 - Build rollout!
MoveBucket-v1-train - (rollout.py:14) - INFO - 2023-04-10,03:46:16 - Rollout environments have seed from [953760095, 953760100)
Segmentation fault (core dumped)

But it was normal when I was training under "PickCube-v0" or "StackCube-v0", so how could I solve the error above?

"Could you please demonstrate the information of each dimension corresponding to the state dictionary inside the obs in the dataset?"

"Could you please demonstrate the information of each dimension corresponding to the state dictionary inside the obs in the dataset?" For example, what information each of the 38 dimensions in the state dictionary of the pickcube environment represents, and what each dimension corresponds to in terms of information.

Issues when loading too many geometries

I tried to import many objects into the simulation by running a load_asset function. However, after loading about 20 geometries, I got the following error:

RuntimeError: Failed to submit geometry because geometries buffer size has been exceeded. .

Is there a simple way to fix this? I did call scene.remove_articulation(obj) after loading each object. The load_asset function is pasted below:

def _load_asset(self, urdf_path):
        loader = self.scene.create_urdf_loader()
        loader.fix_root_link = True
        loader.load_multiple_collisions_from_file = True
        density = 8e3
        obj = loader.load(str(urdf_path), config={"density": density})
        # obj.set_name("faucet")
        set_articulation_render_material(obj, color=hex2rgba("#AAAAAA"), metallic=1, roughness=0.4)
        return obj

Installation fail:

Hi,

I run the pip install command on a newly create conda env with python 3.9:

pip install mani-skill2

But it reports the following error:

========== normal part is ignored =======

error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [1 lines of output]
error in gym setup command: 'extras_require' must be a dictionary whose values are strings or lists of strings containing valid project/version requirement specifiers.
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.

===============================

It seems the required version of gym is not compatible with the new pip version. I resolved this my removing the gym version specification in the requirement file and it seems successfully installed. Not sure if this is a solution for this.

Regarding the environment dimensions

Hi,
When I get the demos from the dataset using this code, the observation and action dimensions are:

dataset = ManiSkill2Dataset(f"demos/rigid_body/{env_id}/trajectory.state.pd_ee_delta_pose.h5")
dataloader = DataLoader(dataset, batch_size=256, num_workers=0, pin_memory=True, drop_last=True, shuffle=True)
obs, action = dataset[0]
print("Observation:", obs.shape)
print("Action:", action.shape)

Observation: torch.Size([42])
Action: torch.Size([7])

However, when I reset the environment, the observation and action dimension looks like 75 and 12.

obs_mode = "state"
env = gym.make("OpenCabinetDrawer-v1", obs_mode=obs_mode)
obs = env.reset() 
obs dimension (75, )

I was wondering if the other 33 dimensions contain some other information. I read through the documents, but couldn't really understand the discrepancy. Could you please let me know about the mismatch?

Adding new robots ?

Hi,
Thanks for making maniskill open. We are really interesting in using it to develop algorithms for robotic manipulation. But we'd like to deploy the policy on some low-cost robot like https://www.trossenrobotics.com/robotic-arms/ros-research-arms.aspx. Gazebo simulation exists for them. We are wondering how many effort will be needed to add them inside maniskill and if you can provide support for that.

Maniskill2 and ManiSkill2-learn on headless GPU nodes in HPC environment?

Hi,

I'm trying to help someone get ManiSkill2 and ManiSkill2-Learn running on our HPC system using GPU nodes. I assume that the general use of the GPUs is for processing but it also appears that there is some expectation of doing some rendering, using a X-Windows display. First, is there a way to decouple these? So that someone can do the GPU processing on a headless GPU node that has no rendering capabilities (we have A100 cards) and then use a different system to do the rendering?

I saw mention of the A100 card in your documentation so I was hopeful. The only JSON file in the /etc/vulkan directory is:

/etc/vulkan/implicit_layer.d/nvidia_layers.json

I saw some mention of running without a DISPLAY variable set so I tried the example:

Singularity> DISPLAY="" python -m mani_skill2.examples.demo_random_action -e PickCube-v0

opts: []
env_kwargs: {}
[2023-06-16 12:54:03.457] [svulkan2] [error] GLFW error: X11: Failed to open display 
[2023-06-16 12:54:03.457] [svulkan2] [warning] Continue without GLFW.
[2023-06-16 12:54:03.459] [svulkan2] [error] Some required Vulkan extension is not present. You may not use the renderer to render, however, CPU resources will be still available.
Segmentation fault (core dumped)

I have also looked through the Issues here on Github but it doesn't seem that the relevant issues have been resolved.

Can you please help me to figure out how to get this going?

Thanks,

Steve

Errors about training with Soft-body

Training issue:
When I run the script:

CUDA_VISIBLE_DEVICES=0 python maniskill2_learn/apis/run_rl.py configs/brl/bc/pointnet_soft_body.py --work-dir ../ManiSkill2-Learn-main/Result/Fill/PPO_pointnet 
--gpu-ids 0 --cfg-options "env_cfg.env_name=Fill-v0" "env_cfg.obs_mode=pointcloud" "env_cfg.control_mode=pd_joint_delta_pos" 
"eval_cfg.num=100" "eval_cfg.num_procs=4" "eval_cfg.save_traj=False" "eval_cfg.save_video=True" 
"replay_cfg.buffer_filenames=../data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-new/soft_body/Fill-v0/trajectory.none.pd_joint_delta_pos_pointcloud.h5"
--resume-from /data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-main-old/maniskill2_learn_pretrained_models_videos/Fill_pcd/model_final.ckpt

the error is:

Fill-v0-train - (run_rl.py:288) - INFO - 2023-06-01,05:06:07 - Train over GPU [0]!
Traceback (most recent call last):
  File "maniskill2_learn/apis/run_rl.py", line 487, in <module>
    main()
  File "maniskill2_learn/apis/run_rl.py", line 452, in main
    run_one_process(0, 1, args, cfg)
  File "maniskill2_learn/apis/run_rl.py", line 432, in run_one_process
    main_rl(rollout, evaluator, replay, args, cfg, expert_replay=expert_replay, recent_traj_replay=recent_traj_replay)
  File "maniskill2_learn/apis/run_rl.py", line 293, in main_rl
    train_rl(
  File "/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-new/maniskill2_learn/apis/train_rl.py", line 185, in train_rl
    all_shape = replay.memory.shape
AttributeError: 'NoneType' object has no attribute 'shape'

When robotiq 140 is used to grasp, the link will separate

Thank you for the great benchmark. I am using the new gripper robotiq 140 provided in tutorials. I found that when I take a grasp action, the links of the gripper will be separated. I checked the urdf as if there were no constraints between certain links. Could you provide some solutions？

ImportError: libvulkan.so.1

from .pysapien import *
ImportError: libvulkan.so.1: cannot open shared object file: No such file or directory

Controller for demo creation

Great work on the benchmarks.

Do you mind releasing the code of the MPC controller used for generating the demonstrations?

The used controller seems to generate pretty jerky trajectories.

Consider this plot showing the actions (x, y, z velocity) for a set of demos for LiftCube.
X-axis is time.

I would like to dampen the controller to generate smoother trajectories, that are better suited to fit GMMs.

Thanks in advance!

Google Colab Visual RL "connection reset by peer", and PPO issues

When running any cell with make_vec_env(), such as the cell with the following code:

env: VecEnv = make_vec_env(
    env_id,
    num_envs,
    obs_mode=obs_mode,
    reward_mode=reward_mode,
    control_mode=control_mode,
    # specify wrappers for each individual environment e.g here we specify the
    # Continuous task wrapper and pass in the max_episode_steps parameter via the partial tool
    wrappers=[
        partial(ContinuousTaskWrapper, max_episode_steps=100)
    ]
)

I get the error:

2023-05-02 18:22:48,424 - mani_skill2 - INFO - RenderServer is running at: localhost:37067
---------------------------------------------------------------------------
ConnectionResetError                      Traceback (most recent call last)
[<ipython-input-24-49bf05d387e1>](https://localhost:8080/#) in <cell line: 44>()
     42 # create num_envs training environments, with max_episode_steps=100
     43 # instead of the default 200 to speed up training
---> 44 env: VecEnv = make_vec_env(
     45     env_id,
     46     num_envs,

5 frames
[/usr/lib/python3.10/multiprocessing/connection.py](https://localhost:8080/#) in _recv(self, size, read)
    377         remaining = size
    378         while remaining > 0:
--> 379             chunk = read(handle, remaining)
    380             n = len(chunk)
    381             if n == 0:

ConnectionResetError: [Errno 104] Connection reset by peer

also in the following cell:

from stable_baselines3 import PPO

set_random_seed(0) # set SB3's global seed to 0
rollout_steps = 3200

# create our model
policy_kwargs = dict(features_extractor_class=CustomExtractor, net_arch=[256, 128])
model = PPO("MultiInputPolicy", env, policy_kwargs=policy_kwargs, verbose=1,
    n_steps=rollout_steps // num_envs, batch_size=400,
    n_epochs=5,
    tensorboard_log="./logs",
    gamma=0.8,
    target_kl=0.2
)

I get the following output/error:

Using cuda device
Wrapping the env with a `Monitor` wrapper
Wrapping the env in a DummyVecEnv.
---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
[<ipython-input-26-b39f3ce5e2d2>](https://localhost:8080/#) in <cell line: 8>()
      6 # create our model
      7 policy_kwargs = dict(features_extractor_class=CustomExtractor, net_arch=[256, 128])
----> 8 model = PPO("MultiInputPolicy", env, policy_kwargs=policy_kwargs, verbose=1,
      9     n_steps=rollout_steps // num_envs, batch_size=400,
     10     n_epochs=5,

6 frames
[/usr/local/lib/python3.10/dist-packages/stable_baselines3/ppo/ppo.py](https://localhost:8080/#) in __init__(self, policy, env, learning_rate, n_steps, batch_size, n_epochs, gamma, gae_lambda, clip_range, clip_range_vf, normalize_advantage, ent_coef, vf_coef, max_grad_norm, use_sde, sde_sample_freq, target_kl, stats_window_size, tensorboard_log, policy_kwargs, verbose, seed, device, _init_setup_model)
    102         _init_setup_model: bool = True,
    103     ):
--> 104         super().__init__(
    105             policy,
    106             env,

[/usr/local/lib/python3.10/dist-packages/stable_baselines3/common/on_policy_algorithm.py](https://localhost:8080/#) in __init__(self, policy, env, learning_rate, n_steps, gamma, gae_lambda, ent_coef, vf_coef, max_grad_norm, use_sde, sde_sample_freq, stats_window_size, tensorboard_log, monitor_wrapper, policy_kwargs, verbose, seed, device, _init_setup_model, supported_action_spaces)
     76         supported_action_spaces: Optional[Tuple[spaces.Space, ...]] = None,
     77     ):
---> 78         super().__init__(
     79             policy=policy,
     80             env=env,

[/usr/local/lib/python3.10/dist-packages/stable_baselines3/common/base_class.py](https://localhost:8080/#) in __init__(self, policy, env, learning_rate, policy_kwargs, stats_window_size, tensorboard_log, verbose, device, support_multi_env, monitor_wrapper, seed, use_sde, sde_sample_freq, supported_action_spaces)
    162         if env is not None:
    163             env = maybe_make_env(env, self.verbose)
--> 164             env = self._wrap_env(env, self.verbose, monitor_wrapper)
    165 
    166             self.observation_space = env.observation_space

[/usr/local/lib/python3.10/dist-packages/stable_baselines3/common/base_class.py](https://localhost:8080/#) in _wrap_env(env, verbose, monitor_wrapper)
    211             if verbose >= 1:
    212                 print("Wrapping the env in a DummyVecEnv.")
--> 213             env = DummyVecEnv([lambda: env])
    214 
    215         # Make sure that dict-spaces are not nested (not supported)

[/usr/local/lib/python3.10/dist-packages/stable_baselines3/common/vec_env/dummy_vec_env.py](https://localhost:8080/#) in __init__(self, env_fns)
     38         VecEnv.__init__(self, len(env_fns), env.observation_space, env.action_space)
     39         obs_space = env.observation_space
---> 40         self.keys, shapes, dtypes = obs_space_info(obs_space)
     41 
     42         self.buf_obs = OrderedDict([(k, np.zeros((self.num_envs, *tuple(shapes[k])), dtype=dtypes[k])) for k in self.keys])

[/usr/local/lib/python3.10/dist-packages/stable_baselines3/common/vec_env/util.py](https://localhost:8080/#) in obs_space_info(obs_space)
     58         dtypes: a dict mapping keys to dtypes.
     59     """
---> 60     check_for_nested_spaces(obs_space)
     61     if isinstance(obs_space, spaces.Dict):
     62         assert isinstance(obs_space.spaces, OrderedDict), "Dict space must have ordered subspaces"

[/usr/local/lib/python3.10/dist-packages/stable_baselines3/common/preprocessing.py](https://localhost:8080/#) in check_for_nested_spaces(obs_space)
    223         for sub_space in sub_spaces:
    224             if isinstance(sub_space, (spaces.Dict, spaces.Tuple)):
--> 225                 raise NotImplementedError(
    226                     "Nested observation spaces are not supported (Tuple/Dict space inside Tuple/Dict space)."
    227                 )

NotImplementedError: Nested observation spaces are not supported (Tuple/Dict space inside Tuple/Dict space).

I'm using the following options:


env_id = "OpenCabinetDrawer-v1"
obs_mode = "rgbd"
control_mode = "base_pd_joint_vel_arm_pd_joint_vel"
reward_mode = "dense"

Error from soft-body environment

I think I was able to build warp with no problems yet I get this error

$ python3 -m mani_skill2.trajectory.replay_trajectory --traj-path demos/soft_body/Excavate-v0/trajectory.h5 --vis
0step [00:00, ?step/s]Warp initialized:
   Version: 0.3.1
   CUDA device: NVIDIA GeForce RTX 3090
   Kernel cache: /home/sean/.cache/warp/0.3.1
2023-06-26 11:00:35,142 - mani_skill2 - INFO - load sdfs from file
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
...
File "/home/sean/.local/lib/python3.8/site-packages/warp_maniskill/warp/context.py", line 1009, in zeros
    raise RuntimeError("Memory allocation failed on device: {} for {} bytes".format(device, num_bytes))
RuntimeError: Memory allocation failed on device: cuda for 120 bytes
0step [00:03, ?step/s]

I'm running it on cuda 11.8 with a 3090.

[Minor] Default observation in some soft body demos seems to be `pointcloud` (leads to different expected behavior).

The current README states for tools/replay_trajectory.py:

We recommend using our script only for converting actions into different control modes and not record any observation information (i.e. without passing --obs-mode). The reason is that (1) some observation modes, e.g. point cloud, can take much space without any post-processing, e.g., point cloud downsampling; in addition, the state mode for soft-body environments also has a similar issue, since the states of those environments are particles. (2) Some algorithms (e.g. GAIL) require custom keys stored in the demonstration files, e.g. next-observation.

However it seems like for 3 of the soft body envs, after loading the demonstrations, the tools/replay_trajectory.py will save using the point cloud type even without --obs-mode explicitly passed on the command line. Thus, by default the code will do the "non recommended" behavior as per the README.

Running this command:

python tools/replay_trajectory.py --traj-path demos/soft_body_envs/Excavate-v0/trajectory.h5   --save-traj --target-control-mode pd_joint_delta_pos --num-procs 10

Will create:

demos/soft_body_envs/Excavate-v0:
total 21G
-rw-rw-r-- 1 seita seita 219M Aug  4 11:28 trajectory.h5
-rw-rw-r-- 1 seita seita  57K Aug  4 11:28 trajectory.json
-rw-rw-r-- 1 seita seita  21G Aug  9 10:59 trajectory.pointcloud.pd_joint_delta_pos.h5
-rw-rw-r-- 1 seita seita  60K Aug  9 10:59 trajectory.pointcloud.pd_joint_delta_pos.json

Whereas for another env like PegInsertionSide-v0 I will see:

demos/rigid_body_envs/PegInsertionSide-v0:
total 97M
-rw-rw-r-- 1 seita seita  49M Aug  4 11:27 trajectory.h5
-rw-rw-r-- 1 seita seita 398K Aug  4 11:27 trajectory.json
-rw-rw-r-- 1 seita seita  48M Aug  9 10:23 trajectory.none.pd_joint_delta_pos.h5
-rw-rw-r-- 1 seita seita 403K Aug  9 10:23 trajectory.none.pd_joint_delta_pos.json

So here we have trajectory.none as the prefix instead of trajectory.pointcloud and the file size is much smaller for the former.

The reason is I think that the trajectory.json files we load has pointcloud as the obs_mode for soft body envs, for example:

(mani_skill2) seita@takeshi:~/ManiSkill2 (main)$ head -n 20 demos/soft_body_envs/Excavate-v0/trajectory.json 
{
  "env_info": {
    "env_id": "Excavate-v0",
    "max_episode_steps": 250,
    "env_kwargs": {
      "obs_mode": "pointcloud",
      "reward_mode": "dense",
      "control_mode": "pd_joint_pos"
    }
  },
  "commit_info": {
    "commit_id": "ef73ec44f5f0f22c9b2f71e682fbdca0c7d377fa",
    "branch": "yihe-dev"
  },
  "episodes": [
    {
      "episode_id": 0,
      "episode_seed": 1,
      "reset_kwargs": {
        "seed": 1,
(mani_skill2) seita@takeshi:~/ManiSkill2 (main)$

The same is true with Fill-v0 and Hang-v0 though interestingly not Pour-v0 (which uses particle for some reason). The rigid body envs have none as the observation mode.

To be clear, this doesn't affect the correctness of the code, though it might be easier (possibly for future users?) if they can use the "none" observation type to save on disk space and for ease of compatibility with the ManiSkill2-Learn commands which seem to expect trajectory.none. Or maybe a possible remark to this in the README could help clarify?

Thanks for the code and environments. :)

How to download the ignored files in the /mani_skill2/assets/ ?

I want to convert the demonstrations using:

python tools/convert_state.py --env-name AssemblingKits-v0 --num-procs 8
--traj-name ../envs/rigid_body_envs/AssemblingKits-v0/trajectory.h5
--json-name ../envs/rigid_body_envs/AssemblingKits-v0/trajectory.json
--output-name ../envs/rigid_body_envs/AssemblingKits-v0/trajectory_out.h5
--control-mode pd_joint_delta_pos --max-num-traj -1 --obs-mode pointcloud
--n-points 1200 --obs-frame base --reward-mode dense

However, the output is :
FileNotFoundError: [Errno 2] No such file or directory: '~/Work/Research/maniskill/ManiSkill2-main/mani_skill2/assets/assembling_kits/kits/kit_232.json'
How can I download these files.

Running soft body envs on multiple GPUs

Hi there! I'm trying to run soft body envs on multiple GPUs, but it's only okay to run on the first card (that card has monitor plugged in). When I'm running on other cards, I did the followings:

export CUDA_VISIBLE_DEVICES=1
python -m mani_skill2.examples.demo_random_action -e Pinch-v0

But I get the following error

RuntimeError: vk::PhysicalDevice::createDeviceUnique: ErrorInitializationFailed

Assets and demos missing?

Hi, here's an error.

python tools/replay_trajectory.py --traj-path demos/rigid_body_envs/TurnFaucet-v0/20220815/5000.h5 --vis

AssertionError: ManiSkill2/mani_skill2/assets/partnet_mobility/dataset/5076/mobility_cvx.urdf is not found. Please download Partnet-Mobility (ManiSkill2022) first

Also, can't find the demonstrations for the rest of the articulated tasks.

Generally, is there any visual variation in the task objcets? (e.g., different cubes/colour faucets like in the shelves from ManiSkill1)

RuntimeError: Cannot find a suitable rendering device

After I install the package, I run the following command:
python -m mani_skill2.examples.demo_random_action

opts: []
env_kwargs: {}
Traceback (most recent call last):
File "/home/ps/anaconda3/envs/maniskill/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/ps/anaconda3/envs/maniskill/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/ps/anaconda3/envs/maniskill/lib/python3.8/site-packages/mani_skill2/examples/demo_random_action.py", line 71, in
main()
File "/home/ps/anaconda3/envs/maniskill/lib/python3.8/site-packages/mani_skill2/examples/demo_random_action.py", line 34, in main
env: BaseEnv = gym.make(
File "/home/ps/anaconda3/envs/maniskill/lib/python3.8/site-packages/gym/envs/registration.py", line 235, in make
return registry.make(id, **kwargs)
File "/home/ps/anaconda3/envs/maniskill/lib/python3.8/site-packages/gym/envs/registration.py", line 129, in make
env = spec.make(kwargs)
File "/home/ps/anaconda3/envs/maniskill/lib/python3.8/site-packages/gym/envs/registration.py", line 87, in make
env = self.entry_point(_kwargs)
File "/home/ps/anaconda3/envs/maniskill/lib/python3.8/site-packages/mani_skill2/utils/registration.py", line 92, in make
env = env_spec.make(kwargs)
File "/home/ps/anaconda3/envs/maniskill/lib/python3.8/site-packages/mani_skill2/utils/registration.py", line 34, in make
return self.cls(_kwargs)
File "/home/ps/anaconda3/envs/maniskill/lib/python3.8/site-packages/mani_skill2/envs/pick_and_place/pick_cube.py", line 22, in init
super().init(*args, **kwargs)
File "/home/ps/anaconda3/envs/maniskill/lib/python3.8/site-packages/mani_skill2/envs/pick_and_place/base_env.py", line 27, in init
super().init(*args, **kwargs)
File "/home/ps/anaconda3/envs/maniskill/lib/python3.8/site-packages/mani_skill2/envs/sapien_env.py", line 102, in init
self._renderer = sapien.SapienRenderer(**renderer_kwargs)
RuntimeError: Cannot find a suitable rendering device

haosulab / maniskill Goto Github PK

maniskill's Introduction

ManiSkill 3 (Beta)

Installation

Getting Started

License

maniskill's People

Contributors

Stargazers

Watchers

Forkers

maniskill's Issues

but get the following error:

Recommend Projects

Recommend Topics

Recommend Org