9DTact

Overview
Installation
3D Shape Reconstruction
6D Force Estimation
Run in ROS

Overview

This repository provides open-source files of the paper:

9DTact: A Compact Vision-Based Tactile Sensor for Accurate 3D Shape Reconstruction and Generalizable 6D Force Estimation
Changyi Lin, Han Zhang, Jikai Xu, Lei Wu, and Huazhe Xu
RAL, 2023
Website / Arxiv Paper / Video Tutorial

@article{lin20239dtact,
  title={9DTact: A Compact Vision-Based Tactile Sensor for Accurate 3D Shape Reconstruction and Generalizable 6D Force Estimation},
  author={Lin, Changyi and Zhang, Han and Xu, Jikai and Wu, Lei and Xu, Huazhe},
  journal={IEEE Robotics and Automation Letters},
  year={2023},
  publisher={IEEE}
}

Installation

Create a conda environment:

conda create -n 9dtact python=3.8

Install pytorch (choose the version that is compatible with your computer):

conda activate 9dtact
conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia

In this repository, install the other requirements:

pip install -e .

3D Shape Reconstruction

For all the terminals used in this section, they are located in the shape_reconstruction directory and based on the 9dtact conda environment:

cd shape_reconstruction
conda activate 9dtact

1. Camera Calibration

Before you start, 3d print the calibration board.
Run:

python3 _1_Camera_Calibration.py

Just follow the printed instructions.

2. Sensor Calibration

Firstly, prepare a ball with a radius of 4.0 mm. (The radius of the ball depends on the thickness of the sensor surface. 4.0 mm is just a recommendation.)
Then, run:

python3 _2_Sensor_Calibration.py

Just follow the printed instructions.

3. Shape Reconstruction

python3 _3_Shape_Reconstruction.py

6D Force Estimation

1. BOTA Driver

If you want to collect force data with a BOTA MiniONE Pro sensor, you need to:
Create a directory named 'bota_ws' as the ROS workspace, and install the bota driver package.

2. Data Collection

At the first terminal, open the BOTA sensor:

cd ~/xxx/bota_ws # Modify 'xxx' to enter the workspace directory
source devel/setup.bash
roslaunch rokubimini_serial rokubimini_serial.launch

At the second terminal, run:

source ~/xxx/bota_ws/devel/setup.bash
cd data_collection
conda activate 9dtact
python3 collect_data.py

At the third terminal, open the 9DTact sensor:

cd shape-force-ros
conda activate 9dtact
python3 _1_Sensor_ros.py

3. Data Processing

Open a terminal, normalize the wrench:

cd data_collection
conda activate 9dtact
python3 wrench_normalization.py  # remember to modify the object_num

At the same terminal, split the data by running:

python3 split_train_test.py

and also:

python3 split_train_test(objects).py

4. Model Training

If using the defaulted parameters, just run:

cd force_estimation
python3 train.py

You may also choose to use Weights and Bias (wandb) by setting use_wandb as True, which helps to track the training performance.

5. Force Estimation

You need to specify a model saved in the 'saved_models' directory as an estimator, by modifying the 'weights' parameters in the force_config.yaml.
After that, run:

cd force_estimation
python3 _1_Force_Estimation.py

Run in ROS

1. Shape Reconstruction in ROS

At the first terminal, open the 9DTact sensor:

cd shape-force_ros
conda activate 9dtact
python3 _1_Sensor_ros.py

At the second terminal, run:

cd shape-force_ros
conda activate 9dtact
python3 _2_Shape_Reconstruction_ros.py

2. Force Estimation in ROS

At the first terminal, open the 9DTact sensor:

cd shape-force_ros
conda activate 9dtact
python3 _1_Sensor_ros.py

At the second terminal, run:

cd shape-force_ros
conda activate 9dtact
python3 _3_Force_Estimation_ros.py

(Optional for visualization) At the third terminal, open the visualization window:

cd force_estimation
conda activate 9dtact
python3 force_visualizer.py

3. Simultaneous Shape Reconstruction and Force Estimation (SSAF) in ROS

At the first terminal, open the force estimator:

cd shape-force_ros
conda activate 9dtact
python3 _3_Force_Estimation_ros.py

At the second terminal, run:

cd shape-force_ros
conda activate 9dtact
python3 _4_Shape_Force_ros.py

0nhc / 9dtact Goto Github PK

9dtact's Introduction

9DTact

Table of contents

Overview

Installation

Create a conda environment:

Install pytorch (choose the version that is compatible with your computer):

In this repository, install the other requirements:

3D Shape Reconstruction

1. Camera Calibration

2. Sensor Calibration

3. Shape Reconstruction

6D Force Estimation

1. BOTA Driver

2. Data Collection

At the first terminal, open the BOTA sensor:

At the second terminal, run:

At the third terminal, open the 9DTact sensor:

3. Data Processing

Open a terminal, normalize the wrench:

At the same terminal, split the data by running:

4. Model Training

5. Force Estimation

Run in ROS

1. Shape Reconstruction in ROS

At the first terminal, open the 9DTact sensor:

At the second terminal, run:

2. Force Estimation in ROS

At the first terminal, open the 9DTact sensor:

At the second terminal, run:

(Optional for visualization) At the third terminal, open the visualization window:

3. Simultaneous Shape Reconstruction and Force Estimation (SSAF) in ROS

At the first terminal, open the force estimator:

At the second terminal, run:

9dtact's People

Contributors

Recommend Projects

Recommend Topics

Recommend Org