ManiSkill2 is a unified benchmark for learning generalizable robotic manipulation skills powered by SAPIEN. It features 20 out-of-box task families with 2000+ diverse object models and 4M+ demonstration frames. Moreover, it empowers fast visual input learning algorithms so that a CNN-based policy can collect samples at about 2000 FPS with 1 GPU and 16 processes on a workstation. The benchmark can be used to study a wide range of algorithms: 2D & 3D vision-based reinforcement learning, imitation learning, sense-plan-act, etc.

Please refer to our documentation to learn more information. There are also hands-on tutorials (e.g, quickstart colab tutorial).

Table of Contents

• Installation
• Getting Started
• Reinforcement Learning Example with ManiSkill2-Learn
• Demonstrations

• Leaderboard
• License
• Citation

From pip:

pip install mani-skill2

From github:

pip install --upgrade git+https://github.com/haosulab/ManiSkill2.git

From source:

git clone https://github.com/haosulab/ManiSkill2.git
cd ManiSkill2 && pip install -e .

A GPU with the Vulkan driver installed is required to enable rendering in ManiSkill2. The rigid-body environments, powered by SAPIEN, are ready to use after installation. Test your installation:

# Run an episode (at most 200 steps) of "PickCube-v0" (a rigid-body environment) with random actions
# Or specify an environment by "-e ${ENV_ID}"
python -m mani_skill2.examples.demo_random_action

Some environments require downloading assets. You can download all the assets by python -m mani_skill2.utils.download_asset all or download task-specific assets by python -m mani_skill2.utils.download_asset ${ENV_ID}. The assets will be downloaded to ./data/ by default, and you can also use the environment variable MS2_ASSET_DIR to specify this destination.

Please refer to our documentation for details on all supported environments. The documentation also indicates which environments require downloading assets.

The soft-body environments are based on SAPIEN and customized NVIDIA Warp, which requires CUDA toolkit >= 11.3 and gcc to compile. Please refer to the documentation for more details about installing ManiSkill2 Warp.

We further provide a docker image (haosulab/mani-skill2) on Docker Hub and its corresponding Dockerfile.

If you encounter any issues with installation, please see the troubleshooting section for common fixes or submit an issue.

Here is a basic example of how to make an Gym/Gymnasium environment and run a random policy.

import gymnasium as gym
import mani_skill2.envs

env = gym.make("PickCube-v0", obs_mode="rgbd", control_mode="pd_joint_delta_pos", render_mode="human")
print("Observation space", env.observation_space)
print("Action space", env.action_space)

obs, reset_info = env.reset(seed=0) # reset with a seed for randomness
terminated, truncated = False, False
while not terminated and not truncated:
    action = env.action_space.sample()
    obs, reward, terminated, truncated, info = env.step(action)
    env.render()  # a display is required to render
env.close()

Each mani_skill2 environment supports different observation modes and control modes, which determine the observation space and action space. They can be specified by gym.make(env_id, obs_mode=..., control_mode=...).

The basic observation modes supported are pointcloud, rgbd, state_dict and state. Please refer to our documentation for information on the observation and control modes available and their details.

Moreover, you can follow the example to interactively play with our environments.

We provide hands-on tutorials about ManiSkill2. All the tutorials can be found here.

• Getting Started: Jupyter Notebook, Colab
• Reinforcement Learning: Jupyter Notebook, Colab
• Imitation Learning: Jupyter Notebook, Colab
• Environment Customization: Jupyter Notebook, Colab
• Advanced Rendering (ray tracing, stereo depth sensor): Jupyter Notebook

See https://sapien.ucsd.edu/docs/latest/ for the tutorials of SAPIEN (the backend of ManiSkill2).

We provide ManiSkill2-Learn, an improved framework based on ManiSkill-Learn for training RL agents with demonstrations to solve manipulation tasks. The framework conveniently supports both point cloud-based and RGB-D-based policy learning, and the custom processing of these visual observations. It also supports many common algorithms (BC, PPO, DAPG, SAC, GAIL). Moreover, this framework is optimized for point cloud-based policy learning, and includes some helpful and empirical advice to get you started.

Please see our documentation for more details.

All rigid body environments in ManiSkill are licensed under fully permissive licenses (e.g., Apache-2.0).

However, the soft body environments will follow Warp's license. Currently, they are licensed under NVIDIA Source Code License for Warp.

The assets are licensed under CC BY-NC 4.0.

If you use ManiSkill2 or its assets and models, consider citing the following publication:

@inproceedings{gu2023maniskill2,
  title={ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills},
  author={Gu, Jiayuan and Xiang, Fanbo and Li, Xuanlin and Ling, Zhan and Liu, Xiqiaing and Mu, Tongzhou and Tang, Yihe and Tao, Stone and Wei, Xinyue and Yao, Yunchao and Yuan, Xiaodi and Xie, Pengwei and Huang, Zhiao and Chen, Rui and Su, Hao},
  booktitle={International Conference on Learning Representations},
  year={2023}
}