Full Body Pose Estimation via Wearable Sensor Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current wearable devices are limited in their ability to accurately estimate full-body pose, which hinders detailed fitness tracking and health monitoring, as they often rely on single-point data collection and lack comprehensive three-dimensional skeletal modeling.
Innovation Solution
A method utilizing multiple wearable devices, including headsets and smartwatches, to obtain POV video data, inertial sensor data, and depth data, which are then processed to generate a 3D mesh model and aligned with a machine learning model for accurate full-body pose estimation, allowing for improved tracking and classification of movements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple wearable devices with sensors are used to capture body movement data, then measurement precision of full-body pose estimation is improved, but device complexity increases
Solution Approach 1:
The system divides the full-body pose estimation task into multiple segments by placing wearable devices at different body locations (head, torso, limbs). Each device captures local movement data independently, which is then fused to reconstruct the complete body pose. This segmentation allows high measurement precision without requiring a single complex capture system.
Solution Approach 2:
The patent merges data from multiple wearable devices (headset with IMU, smartwatch with accelerometer, etc.) into a unified full-body pose representation. By combining measurements from different body parts and synchronizing them through temporal and spatial alignment, the system achieves comprehensive pose estimation that would be impossible with a single device.
2Manufacturing precision
If depth data and multiple camera views are integrated to create 3D skeletal models, then manufacturing precision of pose reconstruction is improved, but processing time increases
Solution Approach 1:
The system performs preliminary actions by pre-aligning wearable devices with known body anatomical landmarks and pre-establishing correspondence between 2D camera views and 3D body parts. This preparation enables rapid 3D reconstruction without time-consuming real-time alignment, as the geometric relationships are already predetermined from the device configurations.
Solution Approach 2:
The patent uses 2D projections from multiple camera views as copies of the 3D body structure. By extracting keypoints from 2D images and depth data, then mapping them to corresponding 3D skeletal locations, the system reconstructs the full-body pose efficiently. This copying approach avoids complex direct 3D scanning while maintaining high reconstruction precision.
Data Source
AI summary
Embodiments are disclosed for full body pose estimation using features extracted from multiple wearable devices. In an embodiment, a method comprises: obtaining point of view (POV) video data and inertial sensor data from multiple wearable devices worn at the same time by a user; obtaining depth data capturing the user's full body; extracting two-dimensional (2D) keypoints from the POV video data; reconstructing a full body 2D skeletal model from the 2D keypoints; generating a three-dimensional (3D) mesh model of the user's full body based on the depth data; merging nodes of the 3D mesh model with the inertial sensor data; aligning respective orientations of the 2D skeletal model and the 3D mesh model in a common reference frame; and predicting, using a machine learning model, classification types based on the aligned 2D skeletal model and 3D mesh model.