Foveated Rendering Image Re-projection for Gaze Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Foveated rendering technologies face challenges due to latency and inaccuracy in gaze point estimation, leading to mismatches between the high-quality rendered image and the actual foveated region, resulting in an unpleasant user experience.
Innovation Solution
A method that involves receiving a first estimation of a predicted gaze point before rendering and a second estimation during rendering, with adjustments made if the difference is below a threshold, allowing for re-projection of the image to accurately match the foveated region, thereby improving the alignment of the high-quality image with the user's gaze point.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If foveated rendering is used to reduce computational workload, then rendering speed is improved, but alignment accuracy between high-quality region and foveated region deteriorates due to latency in gaze point estimation
Solution Approach 1:
The system performs preliminary rendering of the high-quality foveated region based on a first estimated gaze point before the actual display moment. This allows the computationally expensive rendering process to start early, improving rendering speed, while the system prepares to adjust the rendered content based on updated gaze point measurements closer to the display time.
Solution Approach 2:
The system changes the parameters of the rendered image through re-projection transformations. By applying geometric transformations to the already-rendered high-quality region based on the difference between first and second gaze point estimations, the system adjusts the alignment without requiring re-rendering, thus maintaining both speed and accuracy.
2Loss of time
If gaze point estimation is performed earlier to enable rendering, then rendering time is reduced, but estimation accuracy deteriorates due to saccades or blinks occurring during the time delay
Solution Approach 1:
The system performs preliminary rendering based on an initial gaze point estimation to reduce rendering time. However, it acknowledges the reliability issue by implementing a second gaze point measurement closer to the display moment and using re-projection to correct any drift caused by saccades or blinks, thus maintaining reliability despite the time delay.
Solution Approach 2:
The system uses feedback from the second gaze point estimation to adjust the previously rendered image. By comparing the second gaze point measurement with the first estimation and applying re-projection transformations, the system compensates for changes in gaze position that occurred during the rendering process, ensuring the high-quality region remains aligned with the actual foveated region.
3Measurement precision
If image re-projection is performed to correct gaze point mismatch, then alignment accuracy is improved, but computational complexity increases
Solution Approach 1:
The system applies re-projection transformations to adjust the parameters of the rendered image based on the difference between two gaze point estimations. This mathematical transformation corrects the alignment of the high-quality foveated region without requiring complex re-rendering operations, thus improving alignment accuracy while keeping the additional computational complexity relatively low.
4Manufacturing precision
If high-quality rendering is applied to the entire scene, then image quality is improved, but computational resource consumption increases significantly
Solution Approach 1:
The system applies high-quality rendering only to the local foveated region where the user is currently looking, rather than rendering the entire scene at high quality. This is achieved by identifying the foveated region based on gaze point estimation and applying high-resolution rendering only to that specific area, while rendering peripheral regions at lower quality, thus significantly reducing computational resource consumption while maintaining perceptual image quality.
Solution Approach 2:
The system segments the scene into different quality regions based on the user's gaze point. The foveated region is separated and rendered at high quality, while the non-foveated peripheral regions are rendered at lower quality. This segmentation approach allows the system to optimize computational resources by allocating high processing power only where the user can actually perceive detail.
Data Source
Figure 1
Figure 2A~2B
Figure 3
AI summary
Technologies for improving foveated rendering of an image by improving the position of the image to be displayed through image re-projection are disclosed. For example, a method may include receiving a first estimation of a predicted gaze point of a user on a display device that is determined before starting rendering a high-quality portion of the image. The method may further include causing the image to be rendered based on the first estimation of the predicted gaze point. The method may also include receiving a second estimation of the predicted gaze point. The second estimation of the predicted gaze point is determined after rendering of the high-quality portion of the image has started. Responsive to determining that the second estimation of the predicted gaze point is different from the first estimation, the method may include adjusting the rendered image based on the second estimation of the predicted gaze point and transmitting the adjusted image to the display device for display.