Off-Axis Eye Tracking Using Multi-Task Neural Gaze Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing augmented reality systems face challenges in accurately estimating eye gaze and understanding user expressions due to the complexity of human visual perception and the need for energy-efficient rendering without obstructing the user's view.
Innovation Solution
A single deep neural network is trained to perform multiple heterogeneous tasks related to eye gaze estimation and semantic user understanding for off-axis cameras, including eye segmentation, blink detection, emotive expression classification, and pupil and cornea center estimation, using both hand-labeled supervision and model-based supervision.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a single deep neural network is trained to perform multiple heterogeneous tasks (eye segmentation, blink detection, expression classification, pupil and cornea center estimation), then measurement precision and system robustness are improved, but device complexity and computational overhead increase
Solution Approach 1:
The patent implements a single deep neural network that simultaneously performs multiple heterogeneous tasks including eye segmentation, blink detection, emotive expression classification, and pupil and cornea center estimation. This multi-functional network architecture consolidates what would traditionally require separate processing systems into one unified model, improving measurement precision while managing device complexity through shared feature extraction layers.
2Use of energy by moving object
If energy-efficient rendering is implemented to reduce power consumption, then use of energy is improved, but the ability to provide high-quality visual content and accurate gaze estimation may be compromised
Solution Approach 1:
The neural network performs eye segmentation as a distinct task that divides the eye region into meaningful parts (pupil, iris, sclera, etc.). This segmentation capability enables the system to identify and process only the most relevant visual features for gaze estimation, allowing for energy-efficient rendering by focusing computational resources on critical regions rather than processing the entire visual field at high fidelity.
Solution Approach 2:
The system applies local quality processing by focusing detailed analysis on specific eye regions (pupil center, cornea center, iris boundaries) while using coarser processing for other areas. This approach maintains high measurement precision for gaze estimation while reducing overall energy consumption by avoiding uniform high-quality processing across the entire visual scene.
3Ease of operation
If off-axis cameras are used to avoid obstructing the user's view, then ease of operation is improved, but measurement precision and difficulty of detecting and measuring eye gaze increase
Solution Approach 1:
The patent addresses the off-axis camera challenge by incorporating multiple task-specific layers that process eye images from non-frontal angles. The neural network architecture includes specialized components for detecting glints, segmenting eye parts, and estimating gaze direction that are trained specifically on off-axis imagery, thereby maintaining measurement precision while preserving user comfort through unobstructed viewing.
Solution Approach 2:
The system adapts to off-axis camera parameters by training the neural network with transformed eye images that account for varying camera angles, distances, and orientations. The model learns to compensate for perspective distortions, lighting variations, and geometric transformations inherent in off-axis imaging, maintaining accurate gaze estimation despite the challenging viewing geometry.
Data Source
AI summary
Techniques related to the computation of gaze vectors of users of wearable devices are disclosed. A neural network may be trained through first and second training steps. The neural network may include a set of feature encoding layers and a plurality of sets of task-specific layers that each operate on an output of the set of feature encoding layers. During the first training step, a first image of a first eye may be provided to the neural network, eye segmentation data may be generated using the neural network, and the set of feature encoding layers may be trained. During the second training step, a second image of a second eye may be provided to the neural network, network output data may be generated using the neural network, and the plurality of sets of task-specific layers may be trained.