Enabling Hand Pose Tracking with Biomechanical Constraints

Tutian Tang2
Zhenjun Yu2
1Southeast University
2Shanghai Jiao Tong University
*Equal Contribution


This work proposes a novel learning framework for visual hand dynamics analysis that takes into account the physiological aspects of hand motion. The existing models, which are simplified joint-actuated systems, often produce unnatural motions. To address this, we integrate a musculoskeletal system with a learnable parametric hand model, MANO, to create a new model, MS-MANO. This model emulates the dynamics of muscles and tendons to drive the skeletal system, imposing physiologically realistic constraints on the resulting torque trajectories. We further propose a simulation-in-the-loop pose refinement framework, BioPR, that refines the initial estimated pose through a multi-layer perceptron (MLP) network. Our evaluation of the accuracy of MS-MANO and the efficacy of the BioPR is conducted in two separate parts. The accuracy of MS-MANO is compared with MyoSuite, while the efficacy of BioPR is benchmarked against two large-scale public datasets and two recent state-of-the-art methods. The results demonstrate that our approach consistently improves the baseline methods both quantitatively and qualitatively.
Mano model
The parametric MANO model
Musculoskeletal Structure
Musculoskeletal structure of hands
The proposed MS-MANO model




Results 0
Results 1
Results 2
Results 3
Results 2
Results 3



If you have any inquiries or require further clarification regarding the paper, its associated code, or any other related questions, we encourage you to get in touch with us. Please feel free to send your questions via email to xiepf2002 at gmail dot com.


The website is a static site built using Next.js and Tailwind CSS. The source code is available at


  title = {MS-MANO: Enabling Hand Pose Tracking with Biomechanical Constraints},
  author = {Xie*, Pengfei and Xu*, Wenqiang and Tang, Tutian and Yu, Zhenjun and Lu, Cewu},
  booktitle = {{CVPR} IEEE Conference on Computer Vision and Pattern Recognition},
  pages = {},
  year = {2024},