HMD Controller Tracking via Camera and IMU Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current VR and AR systems for mobile terminals lack accurate and convenient 6-DoF tracking systems, leading to reduced immersion and usability, as they either lack tracking systems or require external devices, and there is no system compatible with both 6-DoF head and controller tracking.
Innovation Solution
A method and system using a head-mounted display (HMD) with a camera and inertial measurement unit (IMU) to track a controller's position and rotation by capturing feature points and synchronizing state information, enabling 6-DoF tracking without external devices, suitable for mobile use.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a 6-DoF tracking system is implemented in mobile headsets, then interaction accuracy and immersion are improved, but device complexity and cost increase due to requiring external tracking devices
Solution Approach 1:
The patent combines the camera and IMU into a single integrated HMD device, eliminating the need for separate external tracking devices. The camera captures feature points in the environment while the IMU measures device orientation, and both are processed together to achieve 6-DoF tracking of controllers directly through the HMD without requiring external base stations or cameras.
Solution Approach 2:
The HMD serves multiple functions: it displays virtual content, captures images of the environment through its camera, and processes controller tracking data. The camera in the HMD is used for both environmental mapping and controller feature point detection, while the IMU provides both head orientation and controller orientation data, reducing the need for dedicated tracking hardware.
2Device complexity
If 3-DoF tracking is used in mobile headsets, then device complexity is reduced, but interaction accuracy and immersion deteriorate
Solution Approach 1:
The system dynamically switches between different tracking modes (3-DoF and 6-DoF) based on the detected feature points and environmental conditions. When sufficient feature points are detected, the system activates 6-DoF tracking for high-precision interaction; when feature points are insufficient, it falls back to 3-DoF tracking to maintain functionality, providing adaptive tracking precision.
Solution Approach 2:
The system changes the tracking parameter dimension from 3-DoF to 6-DoF by adding camera-based position estimation to IMU-based orientation estimation. This parameter expansion allows the system to represent controller state more completely (position + orientation) when environmental features permit, thereby improving interaction accuracy without permanently increasing system complexity.
3Measurement precision
If PC with 6DoF tracking system is used, then interaction accuracy is improved, but ease of operation deteriorates due to requiring external devices
Solution Approach 1:
The HMD performs self-contained 6-DoF tracking using its own integrated camera and IMU without requiring external base stations, cameras, or additional equipment. The system autonomously captures feature points, processes controller movements, and maintains tracking accuracy entirely within the portable device, eliminating the need for external PC infrastructure while preserving 6-DoF interaction capability.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
Improves interaction accuracy and immersion by enabling simultaneous 6-DoF head and controller tracking within the HMD, enhancing usability and portability in visual enhancement technologies.
Implementation Method 1
acquiring a feature point of a controller by a camera of a head-mounted display (HMD)
Implementation Method 2
receiving state information of the controller collected by an inertial measurement unit (IMU) of the controller, wherein the state information includes position information and rotation information of the controller
Data Source
AI summary
A tracking method, comprising: acquiring a feature point of a controller by a camera of a HMD, wherein the feature point is used to indicate a corresponding location of the controller; receiving state information of the controller collected by an IMU of the controller, wherein the state information includes position information and rotation information of the controller; and determining a current state of the controller according to the feature point and the state information, wherein the current state includes a current position and a current rotation of the controller.