Light Field Camera Eye Tracking for HMDs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing eye tracking systems in head-mounted displays (HMDs) rely on corneal spherical reflection models and external illumination, limiting their flexibility and accuracy, and require precise positioning of light sources, which can be cumbersome and inefficient.
Innovation Solution
The integration of light field cameras within HMDs that capture plenoptic images, including light intensity and direction data, allows for the creation of a 3D light field model of the user's eye, enabling gaze direction determination without relying on corneal reflection models or external illumination, and providing flexibility in camera placement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional cameras are used for eye tracking, then the system structure is simple, but the tracking accuracy is limited due to lack of light direction information
Solution Approach 1:
The patent transitions from conventional 2D image capture to 4D light field capture by adding spatial and angular dimensions. The light field camera captures not only light intensity but also light ray directions, enabling precise determination of eye gaze direction through depth information and corneal reflection analysis without requiring complex external illumination systems.
Solution Approach 2:
The patent introduces a light field camera as an intermediary device that captures plenoptic images containing both spatial and angular light information. This intermediary captures corneal reflections and eye geometry data, which are then processed to determine gaze direction, replacing the need for complex external illumination sources and simplifying the overall system while improving accuracy.
2Ease of operation
If corneal spherical reflection models are used, then the eye tracking can be performed, but the system requires precise positioning of external illumination sources
Solution Approach 1:
The light field camera performs self-illumination by capturing light reflections from the cornea that naturally occur from the display screen or ambient light. The system uses the light field information to identify corneal reflections and calculate gaze direction without requiring external illumination sources, eliminating the need for precise positioning of such sources while maintaining tracking accuracy.
3Adaptability or versatility
If light field cameras are integrated into HMDs, then flexibility in camera placement is improved, but the device complexity increases
Solution Approach 1:
The light field camera serves multiple functions within the HMD: it captures eye gaze direction, determines eye geometry, tracks corneal reflections, and provides depth information for virtual reality rendering. This multi-functionality justifies the added complexity by eliminating the need for separate illumination sources and multiple cameras, while providing flexible placement options within the HMD structure.
4Use of energy by moving object
If high resolution is maintained across the entire electronic display, then the image quality is high, but the computational resources are excessively consumed
Solution Approach 1:
The system uses light field-based eye tracking to determine the user's gaze direction and dynamically adjusts the rendering resolution based on the foveal region. High resolution rendering is applied only to the small area where the user is currently looking, while peripheral areas are rendered at lower resolution, significantly reducing computational resource consumption while maintaining perceived image quality.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach enables accurate and flexible eye tracking, allowing for dynamic adjustment of display resolution based on user gaze direction, reducing computational resources and enhancing user experience in VR and AR environments.
Implementation Method 1
The one or more light field cameras capture plenoptic images of the user's eyes. A plenoptic image includes light intensity data as well as direction data of the captured light rays.
Implementation Method 2
The HMD may include light sources positioned inside the HMD and that illuminate the user's eyes inside the HMD.
Data Source
AI summary
A head mounted display (HMD) includes one or more light field cameras for tracking one or both eyes of a user wearing the HMD. The HMD optionally includes light sources positioned inside the HMD and that illuminate one or both of the eyes of the user. The light field camera captures a plenoptic image of the user's eye. The plenoptic image includes light intensity data and direction data of the captured light rays. An eye tracking system updates a 3D light field model of the user's eye based on depth information from the plenoptic image frame. The eye tracking system identifies an iris plane of the user's eye using the 3D light field model. The eye tracking system determines a gaze direction of the user's eye by identifying the normal to the center of the iris plane.