Minimal-Latency Tracking for Head-Worn AR Displays
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current augmented reality (AR) systems, particularly optical see-through head-worn displays, face significant latency issues that prevent accurate alignment of virtual and real-world objects, rendering many applications unusable due to delays in tracking, rendering, and display processes.
Innovation Solution
A system and method for minimal-latency tracking and display that utilizes rolling shutter cameras for high-frequency sampling, combined with a global visual mapping approach and a cascade of simpler renderers to achieve low-latency pose estimation and image updates, allowing for real-time alignment of virtual and real-world objects by processing each row of the image independently and updating only the necessary scanlines in the display.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If conventional frame-based processing is used in AR systems, then complete image rendering is achieved, but latency increases to several frames making real-time alignment impossible
Solution Approach 1:
The patent segments the image processing into individual scanlines that can be processed and displayed independently. Instead of waiting for complete frame rendering, each scanline is processed as it becomes ready and displayed immediately, eliminating the need to wait for the entire frame to be rendered. This scanline-by-scanline processing approach reduces latency from multiple frames to sub-frame levels while maintaining alignment accuracy through continuous updates.
Solution Approach 2:
The system performs preliminary calculations and prepares image data in advance where possible, using a cascade of renderers that work in parallel to pre-compute portions of the image. By preparing scanline data before it is needed for display and using predictive algorithms to anticipate required updates, the system minimizes processing delays while maintaining synchronization between real and virtual worlds.
2Measurement precision
If high-frequency sampling and continuous updates are implemented, then alignment accuracy improves, but processing complexity and computational load increase
Solution Approach 1:
The system implements partial action by updating only the portions of the image that have changed or need updating, rather than processing and displaying every scanline continuously. The error image analysis identifies specific regions requiring correction, and only those portions are re-rendered and displayed. This selective updating approach maintains high tracking precision while significantly reducing computational complexity compared to full-frame continuous processing.
Solution Approach 2:
The system uses dynamic processing where the level of detail and update frequency adapts based on scene complexity, motion detection, and error analysis. When the scene is stable, updates are minimized; when motion or errors are detected, processing intensity increases automatically. This dynamic approach allows high precision tracking when needed while reducing overall processing complexity during stable periods.
Data Source
AI summary
Methods, systems, and computer readable media for minimal-latency tracking and display for matching real and virtual worlds in head-worn displays are disclosed. According to one aspect, a method for minimal-latency tracking and display for matching real and virtual worlds in head-worn displays includes calculating a desired image, calculating an error image as the difference between the desired image and an image currently being perceived by a user, identifying as an error portion a portion of the error image having the largest error, updating a portion of a projected image that corresponds to the error portion, and recalculating the image currently being perceived by a user based on the updated projected image.