Late Warping for AR Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Augmented reality (AR) devices experience motion-to-photon latency due to the time it takes to process and render virtual content, which is exacerbated by the movement of both the AR device and physical objects, leading to a diminished user experience.
Innovation Solution
A system and method that applies a late-warping process optimized for both the motion of the AR device and physical objects, using computer vision and inertial data to accurately update virtual content placement in real-time, reducing latency by tracking the latest location of objects and applying a time-warp transformation to the rendered content.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the AR device processes and renders virtual content in real-time, then the user experience is enhanced with accurate virtual content placement, but motion-to-photon latency increases due to the time required for processing and rendering
Solution Approach 1:
The system performs preliminary rendering of virtual content based on the initial pose before the device moves, then applies a time-warp transformation after movement to adjust the rendered content to the updated pose. This preliminary action allows the rendering process to start early while the transformation step corrects for motion, effectively reducing motion-to-photon latency while maintaining placement accuracy.
2Reliability
If the AR device updates virtual content continuously based on device movement, then the illusion of virtual content presence is maintained, but processing complexity and time consumption increase
Solution Approach 1:
The system performs preliminary rendering of virtual content based on the initial pose before the device moves, then applies a time-warp transformation after movement to adjust the rendered content to the updated pose. This preliminary action allows the rendering process to start early while the transformation step corrects for motion, effectively reducing motion-to-photon latency while maintaining placement accuracy.
3Loss of time
If the AR device tracks both device pose and physical object location to reduce latency, then motion-to-photon latency is reduced, but the system complexity and computational requirements increase
Solution Approach 1:
The system uses an intermediary time-warp transformation process that takes the rendered virtual content and adjusts it based on the difference between initial and updated pose. This intermediary step simplifies the tracking requirements by only needing to capture the motion delta rather than continuously tracking and re-rendering all virtual content from scratch, reducing computational complexity while maintaining low latency.
Data Source
AI summary
A method for minimizing latency of moving objects in an augmented reality (AR) display device is described. In one aspect, the method includes determining an initial pose of a visual tracking device, identifying an initial location of an object in an image that is generated by an optical sensor of the visual tracking device, the image corresponding to the initial pose of the visual tracking device. rendering virtual content based on the initial pose and the initial location of the object, retrieving an updated pose of the visual tracking device, tracking an updated location of the object in an updated image that corresponds to the updated pose, and applying a time warp transformation to the rendered virtual content based on the updated pose and the updated location of the object to generate transformed virtual content.