Foveated Rendering Latency Reduction via Saccade Velocity Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current eye-tracking technologies in VR/AR systems face challenges in achieving low latency, especially during fast eye movements like saccades, which affect the accuracy and user experience of foveated rendering.
Innovation Solution
A VR/AR wearable assembly with a photosensor oculography (PSOG) system that includes an eye-tracking module capable of detecting saccades, determining their peak velocity, and rendering foveated images at the final saccade end gaze position, utilizing a processor coupled with a display device and PSOG assembly to provide minimal latency eye position data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional eye-tracking technologies are used to achieve acceptable accuracy, then the rendering system can deliver foveated images, but the system latency remains too high (28 ms) to be unnoticeable during fast eye movements like saccades
Solution Approach 1:
The system performs preliminary actions by detecting saccade onset and predicting the final gaze position before the saccade completes. The eye-tracking module identifies when a saccade begins and uses velocity-based prediction algorithms to determine where the eye will land, allowing the rendering system to prepare the foveated image in advance. This preliminary preparation reduces the perceived latency by having the correct image ready before the eye actually reaches the target position.
Solution Approach 2:
The system dynamically adjusts the rendering pipeline based on real-time eye movement detection. When a saccade is detected, the system transitions from static rendering to dynamic prediction mode, where the gaze position is continuously updated based on eye velocity measurements. This dynamic adaptation allows the system to maintain low latency during fast movements by adjusting the prediction horizon and rendering parameters in real-time.
2Productivity
If the sampling rate is increased to capture fast saccade movements, then the eye position tracking becomes more precise, but the processing complexity and computational load increase
Solution Approach 1:
The system applies local quality by focusing computational resources only when and where needed. During normal fixation periods, standard tracking algorithms are used with lower computational demands. However, when a saccade is detected through velocity thresholding, the system locally intensifies processing by activating prediction algorithms and increasing sampling frequency specifically for that time window. This localized enhancement maintains high accuracy during critical moments without sustaining high processing complexity continuously.
Solution Approach 2:
The system dynamically changes processing parameters based on eye movement state. The sampling rate, prediction horizon, and algorithm complexity are adjusted as parameters depending on whether the eye is in fixation or saccade mode. During saccades, the system increases the sampling rate and activates velocity-based prediction, while during fixations, it uses standard position-based tracking. This parameter adaptation allows high productivity during movements while maintaining manageable overall complexity.
Data Source
AI summary
A virtual reality/augmented reality (VR/AR) wearable assembly is described herein. The VR/AR wearable assembly includes a display device, a photosensor oculography (PSOG) assembly including an eye tracking assembly, and a processor coupled to the PSOG assembly and the display device. The processor including an eye-tracking module configured to execute an algorithm to render computer-generated images on the display device including the steps of detecting a saccade of a corresponding eye of the patient via the eye tracking assembly and determining an initial saccade gaze position location of the corresponding eye associated with the detected saccade, determining a peak velocity of the saccade, determining a final saccade end gaze position based on the determined peak velocity of the saccade, and rendering a foveated image on the display device at an image location corresponding to the determined final saccade end gaze position.