Inside-Out HMD Tracking With Integrated Sensors and Onboard Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing head-mounted displays (HMDs) for virtual and mixed reality applications are limited by the need for external tracking components that restrict user mobility and require complex setup, leading to tracking inaccuracies and reduced immersion due to narrow viewing angles and slow, inaccurate tracking functions.
Innovation Solution
An integrated HMD that incorporates RGB and mono camera sensors, an inertial measurement unit (IMU), a time-of-flight (ToF) camera, and a speckle pattern projector, enabling inside-out positional, user body, and environment tracking with low latency and high accuracy, allowing for compact, user-friendly operation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If external tracking components (IR cameras, base stations) are used for positional tracking, then tracking accuracy can be achieved, but user mobility is restricted and setup complexity increases
Solution Approach 1:
Instead of using external components to track the HMD (outside-in approach), the patent integrates tracking components inside the HMD to perform inside-out tracking. The IMU, ToF camera, and RGB camera are embedded within the HMD to capture environment features and calculate head position and orientation relative to the environment, reversing the traditional tracking architecture.
Solution Approach 2:
The patent combines multiple tracking functions (positional tracking, body tracking, environment tracking) into a single integrated HMD device. The IMU, ToF camera, RGB camera, and processing unit are merged into one device, eliminating the need for separate external tracking systems and enabling free user mobility throughout the environment.
2Measurement precision
If external tracking components are used, then positional tracking can be achieved, but device complexity and setup requirements increase
Solution Approach 1:
The patent integrates the IMU, ToF camera, RGB camera, and processing unit into a single HMD device, consolidating multiple tracking functions into one unit. This reduces system complexity by eliminating separate external tracking components and simplifying the overall system architecture.
Solution Approach 2:
The HMD performs tracking autonomously using its own integrated sensors and processing unit. The device captures environment features through its cameras, processes the data internally, and calculates head position and orientation without requiring external base stations or cameras, making the system self-sufficient.
3Volume of moving object
If narrow field of view displays are used, then device size can be reduced, but immersion quality decreases
Solution Approach 1:
The system dynamically adjusts the virtual environment based on real-time head tracking data from the IMU and camera sensors. As the user moves their head, the virtual scene updates accordingly to maintain a consistent and immersive experience, allowing for a wider effective field of view without increasing physical device size.
4Device complexity
If slow or inaccurate tracking functions are used, then device complexity can be reduced, but user comfort and immersion decrease due to simulation sickness
Solution Approach 1:
The system continuously monitors head position and orientation through the IMU and camera sensors, providing real-time feedback to the rendering engine. This feedback loop ensures that the virtual environment updates synchronously with user head movements, maintaining accurate tracking and preventing simulation sickness without requiring overly complex tracking hardware.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
Enables high-resolution, low-latency virtual and mixed reality experiences with enhanced mobility and immersion by integrating all tracking components within the HMD, reducing setup complexity and maintaining accurate tracking across large environments.
Implementation Method 1
a time of flight (ToF) camera sensor with an associated IR emitter
Implementation Method 2
an inertial measurement unit (IMU)
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
A Head-Mounted Display system together with associated techniques for performing accurate and automatic inside-out positional, user body and environment tracking for virtual or mixed reality are disclosed. The system uses computer vision methods and data fusion from multiple sensors to achieve real-time tracking. High frame rate and low latency is achieved by performing part of the processing on the HMD itself.