This repository is an official implementation of the AAAI 2023 paper "Two Heads are Better than One: Image-Point Cloud Network for Depth-Based 3D Hand Pose Estimation".

Two Heads are Better than One: Image-Point Cloud Network for Depth-Based 3D Hand Pose Estimation

Pengfei Ren, Chenyu Chen, Jiachang Hao, Haifeng Sun, Qi Qi, Jingyu Wang, Jianxin Liao

• Python >= 3.8
• PyTorch >= 1.10
• CUDA (tested with cuda11.3)
• Other dependencies described in requirements.txt
• Install point operation

  pip install pointnet2_ops_lib/.

• Install Manopth

• Go to MANO website
• Download Models and Code (the downloaded file should have the format mano_v*_*.zip).
• unzip and copy the models/MANO_RIGHT.pkl into the MANO folder
• Your folder structure should look like this:

code/
  MANO/
    MANO_RIGHT.pkl

• Download and decompress DexYCB
• Modify the root_dir in config.py according to your setting.
• Generate json file for data loading (dataloader/DEXYCB2COCO.py)
• In order to speed up the training, you need to generate the hand mesh corresponding to each image according to the MANO annotation.
• Your folder structure should look like this:

DexYCB/
  mesh/
    20200709-subject-01/
        20200709_153548/
            932122062010/
                mesh_000000.txt
                ...
    ...
  20200709-subject-01/
  20200813-subject-02/
  ...
            

• Download and decompress NYU
• Modify the root_dir in config.py according to your setting.

python train_ho.py

python train.py

Remember to change the dataset name in config.py accordingly.