Gaze-Adjusted Image Rendering for XR Motion-to-Photon Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for mitigating motion-to-photons latency in extended reality (XR), virtual reality (VR), and augmented reality (AR) head-mounted displays suffer from inaccuracies in pose prediction, disocclusion issues, and the need for complex reprojection techniques, leading to suboptimal user experiences and discomfort.
Innovation Solution
A method that predicts a user's pose and gaze location, renders an image frame based on these predictions, adjusts the frame to match the actual gaze location, and displays the adjusted frame, utilizing techniques like 3 DOF, 6 DOF, and 9 DOF warping to minimize latency and enhance immersion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If pose-based reprojection methods are used to address motion-to-photons latency, then latency is reduced, but inaccuracies in pose prediction and disocclusion issues arise
Solution Approach 1:
The system performs preliminary actions by predicting future pose and gaze location before rendering the image frame. The processor predicts a future pose and future gaze location, then renders the image frame based on these predictions. This allows the system to compensate for motion-to-photons latency by preparing the rendering in advance, while the gaze prediction mechanism helps maintain accuracy by focusing on where the user will be looking rather than relying solely on pose prediction.
Solution Approach 2:
The system implements feedback by determining an actual gaze location after rendering and using this information to adjust the rendered image frame. The processor determines an actual gaze location and adjusts the rendered image frame accordingly, creating a closed-loop system that corrects for prediction errors and maintains reliability while reducing latency.
2Manufacturing precision
If complex reprojection techniques are applied to adjust displayed image, then image alignment is improved, but device complexity increases
Solution Approach 1:
The system applies local quality by adjusting only the specific region of the rendered image frame that corresponds to the predicted gaze location. Instead of applying complex reprojection techniques to the entire image, the processor adjusts only the region of interest based on the predicted gaze location, reducing computational complexity while maintaining precise alignment in the critical area.
Solution Approach 2:
The system uses preliminary action by predicting the future gaze location before rendering, which simplifies the adjustment process. Rather than performing complex post-rendering reprojection, the system prepares the rendering in advance based on predicted gaze, reducing the complexity of subsequent adjustments while maintaining precision.
3Productivity
If pose prediction is used to predict user gaze, then rendering optimization is achieved, but errors accumulate throughout the reprojection process
Solution Approach 1:
The system combats error accumulation by implementing feedback through gaze detection. The processor determines an actual gaze location after rendering and uses this feedback to adjust the rendered image frame. This closed-loop approach corrects for errors in pose prediction by directly measuring where the user is actually looking, maintaining reliability while preserving rendering optimization efficiency.
Solution Approach 2:
The system uses preliminary action by predicting future gaze location before rendering, which optimizes rendering efficiency. The processor predicts a future gaze location and renders the image frame accordingly, improving productivity. The accuracy is maintained through subsequent gaze-based adjustment, separating the optimization phase from the accuracy verification phase.
Data Source
AI summary
Disclosed is a method of optimizing image rendering implemented in at least one display apparatus that includes predicting a pose and a first gaze location of a user for a display target time; rendering an image frame based on the predicted pose and the first gaze location; determining a second gaze location after rendering of the image frame; adjusting the rendered image frame such that an image area of the rendered image frame under the first gaze location and its surrounding region is moved to the second gaze location; and displaying the adjusted image frame on the at least one display apparatus.