XR4ED publication accepted at ACM Transactions on Graphics

This paper discusses the development of a novel solution called “SparsePoser” for accurately and reliably reconstructing human motion in the context of Virtual Reality (VR) and entertainment applications. The paper addresses the growing demand for cost-effective ways to create full-body animations that can rival the quality of commercial motion capture systems, particularly in light of the rising popularity of the Metaverse and social applications. 

The main challenges this paper focuses on are minimizing the number of sensors attached to a subject’s body while still achieving high-quality full-body pose reconstructions. Sparse data, which arises when using a reduced set of sensors, presents difficulties in accurately reconstructing a person’s full-body pose, leading to issues such as positional drift and pose ambiguity. The paper highlights that although some VR systems offer 6-degree-of-freedom (6-DoF) tracking devices, most existing solutions for reconstructing full-body poses still rely on traditional inverse kinematics (IK) techniques, which can produce non-continuous and unnatural poses. In response to these challenges, the authors introduce SparsePoser, a deep learning-based solution. The core components of SparsePoser include a convolutional-based autoencoder that learns from motion capture data to synthesize high-quality, continuous human poses. Additionally, a learned IK component, comprising lightweight feed-forward neural networks, is used to adjust the hands and feet positions to match the corresponding tracking devices. To validate the effectiveness of their approach, the paper provides extensive evaluation results based on publicly available motion capture datasets and real-time live demonstrations.

 The authors demonstrate that SparsePoser outperforms state-of-the-art techniques that rely on Inertial Measurement Unit (IMU) sensors or 6-DoF tracking devices. Importantly, the system is capable of accommodating users with varying body dimensions and proportions, making it a versatile and robust solution for human motion reconstruction in the context of VR and entertainment applications.