CVPR'24

MusculoSkeletal-MANO
Enabling Hand Pose Tracking with Biomechanical Constraints

Tutian Tang2
Zhenjun Yu2
1Southeast University
2Shanghai Jiao Tong University
*Equal Contribution

Abstract

This work proposes a novel learning framework for visual hand dynamics analysis that takes into account the physiological aspects of hand motion. The existing models, which are simplified joint-actuated systems, often produce unnatural motions. To address this, we integrate a musculoskeletal system with a learnable parametric hand model, MANO, to create a new model, MS-MANO. This model emulates the dynamics of muscles and tendons to drive the skeletal system, imposing physiologically realistic constraints on the resulting torque trajectories. We further propose a simulation-in-the-loop pose refinement framework, BioPR, that refines the initial estimated pose through a multi-layer perceptron (MLP) network. Our evaluation of the accuracy of MS-MANO and the efficacy of the BioPR is conducted in two separate parts. The accuracy of MS-MANO is compared with MyoSuite, while the efficacy of BioPR is benchmarked against two large-scale public datasets and two recent state-of-the-art methods. The results demonstrate that our approach consistently improves the baseline methods both quantitatively and qualitatively.
Mano model
The parametric MANO model
Musculoskeletal Structure
Musculoskeletal structure of hands
MS-MANO
The proposed MS-MANO model

Pipeline

pipeline

Results

Results 0
Results 1
Results 2
Results 3
Results 2
Results 3

Video

Contact

If you have any inquiries or require further clarification regarding the paper, its associated code, or any other related questions, we encourage you to get in touch with us. Please feel free to send your questions via email to xiepf2002 at gmail dot com.

Website

The website is a static site built using Next.js and Tailwind CSS. The source code is available at
github/panoanx/msmano-next

BibTeX

@inproceedings{msmano,
  title = {MS-MANO: Enabling Hand Pose Tracking with Biomechanical Constraints},
  author = {Xie*, Pengfei and Xu*, Wenqiang and Tang, Tutian and Yu, Zhenjun and Lu, Cewu},
  booktitle = {{CVPR} IEEE Conference on Computer Vision and Pattern Recognition},
  pages = {},
  year = {2024},
}