bill007bill / avatarposer Goto Github PK

Jiaxi Jiang¹, Paul Streli¹, Huajian Qiu¹, Andreas Fender¹, Larissa Laich², Patrick Snape², Christian Holz¹

¹ Sensing, Interaction & Perception Lab, Department of Computer Science, ETH Zürich, Switzerland
² Reality Labs at Meta, Zurich, Switzerland

Today's Mixed Reality head-mounted displays track the user's head pose in world space as well as the user's hands for interaction in both Augmented Reality and Virtual Reality scenarios. While this is adequate to support user input, it unfortunately limits users' virtual representations to just their upper bodies. Current systems thus resort to floating avatars, whose limitation is particularly evident in collaborative settings. To estimate full-body poses from the sparse input sources, prior work has incorporated additional trackers and sensors at the pelvis or lower body, which increases setup complexity and limits practical application in mobile settings. In this paper, we present AvatarPoser, the first learning-based method that predicts full-body poses in world coordinates using only motion input from the user's head and hands. Our method builds on a Transformer encoder to extract deep features from the input signals and decouples global motion from the learned local joint orientations to guide pose estimation. To obtain accurate full-body motions that resemble motion capture animations, we refine the arm joints' positions using an optimization routine with inverse kinematics to match the original tracking input. In our evaluation, AvatarPoser achieved new state-of-the-art results in evaluations on large motion capture datasets (AMASS). At the same time, our method's inference speed supports real-time operation, providing a practical interface to support holistic avatar control and representation for Metaverse applications.

• Datasets
• Training
• Testing
• Pretrained Models
• Citation
• License and Acknowledgement

1. Please download the datasets BMLrub, CMU, and HDM05 from AMASS.
2. Download the required body model and placed them in support_data/body_models directory of this repository. For SMPL+H body model, download it from http://mano.is.tue.mpg.de/. Please download the AMASS version of the model with DMPL blendshapes. You can obtain dynamic shape blendshapes, e.g. DMPLs, from http://smpl.is.tue.mpg.de
3. (Optional) If you want to have new random data split, run generate_split.py
4. Run prepare_data.py to preprocess the input data for faster training. The data split for training and testing data in our paper is stored under the folder data_split.

For training, please run:

python main_train_avatarposer.py -opt options/train_avatarposer.json

For testing, please run:

python main_test_avatarposer.py

Click Pretrained Models to download our pretrained model for AvatarPoser, and put it into model_zoo.

If your find our paper or codes useful, please cite our work:

@inproceedings{jiang2022avatarposer,
  title={AvatarPoser: Articulated Full-Body Pose Tracking from Sparse Motion Sensing},
  author={Jiang, Jiaxi and Streli, Paul and Qiu, Huajian and Fender, Andreas and Laich, Larissa and Snape, Patrick and Holz, Christian},
  booktitle={Proceedings of European Conference on Computer Vision},
  year={2022},
  organization={Springer}
}

This project is released under the MIT license. We refer to the code framework in FBCNN and KAIR for network training.