Predictive Eye Tracking for Foveated Rendering in Head-Mounted Displays
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Head-mounted displays (HMDs) face challenges in rendering immersive virtual, augmented, and mixed reality content due to vergence and accommodation conflicts, leading to visual fatigue and inefficiencies in resource utilization, particularly in accurately focusing images on the fovea and managing resolution across the retina.
Innovation Solution
The implementation of predictive eye tracking systems that adjust the display to anticipate future gaze locations and vergence planes, allowing for dynamic rendering of foveated and variable focus images, optimizing resource usage and reducing user discomfort by ensuring high-resolution images are centered on the user's gaze and adjusting focus accordingly.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If conventional displays present images at constant resolution, then the entire image field is rendered uniformly, but resources are wasted on rendering high-resolution images in peripheral regions where the retina cannot perceive high resolution
Solution Approach 1:
The patent applies local quality by rendering different portions of the image at different resolutions based on the user's gaze location. The foveal region (center of gaze) is rendered at high resolution while peripheral regions are rendered at lower resolution, matching the non-uniform sensitivity of the human retina and reducing unnecessary rendering resources.
Solution Approach 2:
The patent uses eye tracking to predict future gaze locations and pre-renders images for those predicted locations before the user actually looks there. This preliminary action allows the system to prepare high-resolution content in advance, reducing latency and ensuring smooth transitions as the user moves their gaze.
2Productivity
If HMDs render images for the entire field of view at high resolution, then complete visual coverage is provided, but processing time and computational resources are excessively consumed
Solution Approach 1:
The system dynamically adjusts rendering quality based on spatial location relative to the user's gaze. Only the foveal region receives high-resolution rendering while peripheral regions use lower resolution, significantly reducing the total number of pixels that need to be processed at high quality while maintaining perceived visual information where it matters most.
Solution Approach 2:
The patent implements partial action by rendering only the necessary portion of the visual field at high resolution (the foveal region) rather than the entire field of view. This partial rendering approach provides sufficient visual information for the user's actual perception needs while avoiding the excessive computational burden of rendering the complete field at uniform high resolution.
3Measurement precision
If the display updates images in real-time based on current gaze location, then visual accuracy is maintained, but latency causes the gaze location to shift during rendering
Solution Approach 1:
The system performs preliminary action by predicting future gaze locations based on current eye movement data and pre-rendering images for those predicted locations before the user actually looks there. This anticipatory rendering eliminates the latency problem by ensuring the correct high-resolution image is ready when the user's gaze arrives at that location.
Solution Approach 2:
The patent implements feedback by continuously monitoring eye tracking data and using it to update and refine gaze location predictions in real-time. This closed-loop feedback system allows the rendering to stay synchronized with actual user gaze movements, maintaining measurement precision while compensating for rendering delays through predictive adjustments.
4Adaptability or versatility
If variable focus display features are integrated with foveated display features, then vergence and accommodation conflicts are addressed, but system complexity increases
Solution Approach 1:
The patent merges variable focus display features with foveated display features into a unified system. The eye tracking data drives both the foveated rendering (spatially selective resolution) and the variable focus adjustment (temporally selective focus) simultaneously, allowing the system to address vergence-accommodation conflicts while maintaining a coordinated approach rather than separate independent systems.
Solution Approach 2:
The eye tracking module serves multiple functions: it provides gaze location data for foveated rendering, predicts future gaze locations for pre-rendering, and determines vergence information for variable focus adjustment. This multi-functionality reduces overall system complexity by using a single data acquisition system to drive multiple display adaptation features rather than requiring separate sensing systems for each function.
Data Source
AI summary
Various aspects of the subject technology relate to prediction of eye movements of a user of a head-mountable display device. Predictive foveated display systems and methods, using the predicted eye movements are also disclosed. Predictive variable focus display systems and methods using the predicted eye movements are also disclosed. Predicting eye movements may include predicting a future gaze location and/or predicting a future vergence plane for the user's eyes, based on the current motion of one or both of the user's eyes. The predicted gaze location may be used to pre-render a foveated display image frame with a high-resolution region at the predicted gaze location. The predicted vergence plane may be used to modify an image plane of a display assembly to mitigate or avoid a vergence/accommodation conflict for the user.