Late Warping for AR Motion-to-Photon Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Augmented reality (AR) devices experience motion-to-photon latency due to the time it takes to process and render virtual content, which is exacerbated by the movement of both the AR device and physical objects, leading to a diminished user experience.
Innovation Solution
A system and method that applies a late-warping process optimized for both the movement of the AR device and physical objects, using computer vision and inertial data to accurately place virtual content in real-time, reducing latency by tracking the motion of physical objects and applying a time-warp transformation to the rendered content.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the AR device processes and renders virtual content in real-time, then the user experience is improved, but motion-to-photon latency increases due to the time required to track device movement, track physical object movement, process environmental data, and render virtual content
Solution Approach 1:
The system performs preliminary tracking of both the AR device and physical objects continuously, maintaining ready-state data about device pose and object locations before rendering is needed. This allows the rendering process to start immediately when virtual content needs to be displayed, reducing the overall motion-to-photon latency while maintaining accurate positioning.
Solution Approach 2:
The system introduces a coordinate transformation intermediary that maps virtual content positions from the device's coordinate system to the physical object's coordinate system. This intermediary layer enables efficient calculation of virtual content placement by leveraging pre-tracked object positions, reducing the computational time needed for real-time rendering while maintaining accuracy.
2Device complexity
If the system tracks only the AR device movement, then the processing complexity is reduced, but the accuracy of virtual content placement deteriorates when physical objects move
Solution Approach 1:
The tracking system is segmented into two independent components: one tracking the AR device movement and another tracking physical object movement. This segmentation allows each component to be optimized separately - the device tracker can use simple inertial sensors while the object tracker uses computer vision, reducing overall complexity while maintaining high placement accuracy through coordinated use of both tracks.
Solution Approach 2:
The system merges the device movement track and physical object movement track into a unified coordinate transformation process. By combining these two tracking streams, the system achieves accurate virtual content placement that accounts for both device and object motion, while the modular architecture keeps the overall system complexity manageable through standardized integration interfaces.
Data Source
AI summary
A method for minimizing latency of moving objects in an augmented reality (AR) display device is described. In one aspect, the method includes determining an initial pose of a visual tracking device, identifying an initial location of an object in an image that is generated by an optical sensor of the visual tracking device, the image corresponding to the initial pose of the visual tracking device, rendering virtual content based on the initial pose and the initial location of the object, retrieving an updated pose of the visual tracking device, tracking an updated location of the object in an updated image that corresponds to the updated pose, and applying a time warp transformation to the rendered virtual content based on the updated pose and the updated location of the object to generate transformed virtual content.