Eye-Tracked 360 Video Rendering with Foveal Resolution Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing display technologies face challenges in efficiently managing eye accommodation and providing immersive viewing experiences in 360 video capture and display systems, particularly in applications like 3D displays and virtual reality, due to substantial encoding, processing, and power requirements.
Innovation Solution
The use of eye tracking for controlling 360 video capture and display, combined with gaze-based warnings and eye accommodation matching, to optimize video resolution and focus based on viewer interaction, reducing computational load and enhancing immersion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high resolution video is displayed across the entire field of view in 360 video systems, then image quality is improved, but encoding complexity and processing requirements increase substantially
Solution Approach 1:
The patent applies local quality by encoding different regions of the 360 video at different resolutions based on eye tracking data. The foveal region (where the user is looking) is encoded at high resolution, while peripheral regions are encoded at lower resolutions. This resolves the contradiction by maintaining high image quality only where needed, thereby reducing overall encoding complexity and processing requirements.
2Measurement precision
If high resolution video is processed and displayed, then viewing quality is improved, but power consumption increases
Solution Approach 1:
The patent implements partial action by processing and displaying high resolution video only in the foveal region rather than across the entire field of view. The eye tracking system identifies the user's gaze point, and rendering is performed at high quality only in this limited area. This reduces power consumption significantly while maintaining perceived viewing quality, as the human eye is most sensitive to detail in the central vision area.
3Adaptability or versatility
If the entire field of view is rendered at high resolution, then immersion is improved, but processing time and computational load increase
Solution Approach 1:
The patent applies dynamics by making the resolution allocation adaptive and dynamic based on real-time eye tracking data. As the user moves their gaze, the high-resolution rendering region dynamically follows the gaze point. This maintains immersion because the high-quality content always appears where the user is looking, while simultaneously improving processing efficiency by limiting high-resolution rendering to a small moving region rather than the static entire field of view.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
Systems and methods may provide for capturing 360 degree video, and multi-resolution encoding, processing and displaying of the video based on a field of view (FOV) and region of interest (ROI) for a viewer. The ROI may be determined based on eye tracking information (ETI) and the video may be encoded for viewports within the FOV at a high resolution and for other viewports outside the FOV at a lower resolution. ROI in the video may be encoded at a high resolution and areas outside of the ROI may be encoded at a lower resolution. The ETI enables the selective display of one or more warnings based on the gaze of a user to improve the efficiency of the warning. 3D glasses having variable lens may be used to adjust the focal distance of a virtual display to match a virtual distance of an object based on stereo distance cues.