Self-Tracked Controllers for Full-Body Pose Beyond HMD Visibility
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AR/VR systems struggle to estimate a user's full body pose accurately due to HMD cameras being unable to capture the lower body parts, leading to incomplete pose estimation and negatively impacting user experience.
Innovation Solution
Utilizing self-tracking controllers with integrated cameras and IMUs to perform SLAM for self-localization, capturing images of body parts not visible to HMD cameras, and combining these with HMD camera data to determine a full body pose using inverse-kinematic optimizers and muscular-skeletal models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If HMD cameras are used to estimate body pose, then the system structure is simple, but the lower body parts cannot be captured resulting in incomplete pose estimation
Solution Approach 1:
The system divides the body tracking function into multiple segments: HMD cameras capture upper body pose, while controllers capture hand poses and lower body poses. This segmentation allows each component to focus on specific body regions, achieving complete full-body tracking without requiring a single complex camera system to capture everything.
Solution Approach 2:
Controllers serve as intermediary devices between the user and the tracking system. They capture images of body parts not visible to HMD cameras and provide additional pose information, acting as a mediator that bridges the gap between limited HMD camera coverage and complete body pose estimation.
2Loss of information
If controllers with cameras are introduced to capture lower body parts, then complete body pose estimation is achieved, but the device complexity increases
Solution Approach 1:
Controllers perform multiple functions: they track hand poses through IMU sensors, capture images of lower body parts through integrated cameras, and provide visual feedback to users. This multi-functionality reduces the need for separate dedicated devices for each tracking function, thereby managing complexity while achieving comprehensive body pose estimation.
Solution Approach 2:
The system merges the tracking functionality into existing controllers that users already hold. By combining hand tracking capabilities with lower body capture and visual feedback functions in a single device, the system avoids adding separate complex hardware components while achieving complete body pose estimation.
3Measurement precision
If multiple data sources are integrated for full body pose estimation, then measurement accuracy improves, but data processing complexity increases
Solution Approach 1:
The data processing system segments pose estimation into distinct modules: upper body pose estimation from HMD camera images, hand pose estimation from controller IMU data, and lower body pose estimation from controller camera images. Each module processes specific data types independently before integration, reducing overall processing complexity while maintaining high accuracy.
Solution Approach 2:
The system employs feedback mechanisms where the processed pose information is used to adjust and refine the estimation algorithms. Visual feedback from controllers and iterative optimization of pose estimates based on multiple data sources enable accurate full-body tracking while managing processing complexity through adaptive refinement rather than exhaustive computation.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
Accurately estimates the full body pose of users, even when parts are not visible to HMD cameras, enhancing user experience by providing complete body avatars and ensuring privacy through local data processing.
Implementation Method 1
a head pose determined using an inertial measurement unit associated with the headset
Implementation Method 2
the pose of the controller may be determined using simultaneous localization and mapping (SLAM) for self-localization
Data Source
Figure 1A
Figure 1B
Figure 2A
AI summary
In one embodiment, a computing system may determine a pose of a controller held by a user based on sensor data captured by the controller. The system may determine positions of a first set of keypoints associated with a first portion of a body of the user based on images captured by the controller cameras and (2) the controller pose. The system may determine a pose of a headset worn by the user based on sensor data captured by the headset. The system may determine positions of a second set of keypoints associated with a second portion of the body of the user based on images captured by the headset cameras and (2) the pose of the headset. The system may determine a full body pose of the user based at least on the positions of the first set and the second set of keypoints.