Time Warp Engine for AR/MR Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current augmented reality (AR) and mixed reality (MR) systems face significant challenges with motion-to-photon latency, leading to noticeable delays and discrepancies between real-world and virtual graphics, which are exacerbated by the need for higher frame rates and lower latency compared to virtual reality (VR) systems, resulting in issues like 'judder' and 'drifting' effects.
Innovation Solution
Implementing time warp independently for each individual color-frame or pixel-group at a frequency matching the refresh rate of AR/MR systems, significantly increasing the time warp frequency to reduce motion-to-photon latency to less than 6 milliseconds, and using a multi-stage approach to apply time warp re-projections for each sequential visual unit, thereby improving motion estimation accuracy and reducing latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If conventional time warp is applied to the entire image at VR refresh rates (60-90 Hz), then device complexity is reduced, but motion-to-photon latency is too high (>20ms) for AR/MR requirements
Solution Approach 1:
The patent divides the image into multiple independently processable segments (tiles or regions) that can be time-warped simultaneously. This segmentation enables parallel processing of multiple image portions at higher frequencies, reducing overall motion-to-photon latency while maintaining manageable device complexity through distributed processing.
2Loss of time
If time warp frequency is increased to match AR/MR refresh rates (480-720 Hz or higher), then motion-to-photon latency is reduced to <6ms, but device complexity and processing burden increase significantly
Solution Approach 1:
By segmenting the image into multiple tiles that can be processed independently and in parallel, the system achieves high time warp frequencies without proportionally increasing overall device complexity. Each tile can be processed by separate processing units simultaneously.
Solution Approach 2:
The patent applies time warp selectively to specific image regions or tiles rather than requiring complete processing of the entire image at maximum frequency. This partial action approach reduces the total processing burden while still achieving the required latency performance for the most critical visual areas.
3Loss of information
If conventional time warp extrapolates graphics in occluded areas, then no-data regions are filled, but drifting effects occur causing separation between real and simulated objects
Solution Approach 1:
The patent applies different handling strategies to different regions of the image based on local characteristics. In regions where real-world content is visible, the system preserves the original content without extrapolation, while only filling truly necessary gaps. This local differentiation prevents the drifting effect by avoiding uniform extrapolation across the entire image.
Data Source
AI summary
A device, system, and method for applying time warp in an augmented or mixed reality environment. An image may be rendered corresponding to an estimated head position calculated based on head position and orientation measured at the initial time. Time warp may be independently applied to each sequential one of the plurality of segmented visual units of an image (e.g., color-frames or pixel-groups) to generate a plurality of respective time warp visual units corresponding to a plurality of different respective estimated head position and orientations calculated based on head position and orientations measured at a plurality of different respective sequential times after the initial time. The plurality of time warp visual units may be projected in the sequential order at the augmented or mixed reality device.