Off-Axis Eye Tracking Using Multi-Task Gaze Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing augmented reality systems face challenges in accurately estimating eye gaze and understanding user expressions in a manner that is energy-efficient and non-obtrusive, particularly in head-mounted VR, MR, and AR devices, due to the complexity of human visual perception and the difficulty of imaging eyes off-axis.
Innovation Solution
A single deep neural network is trained to perform multiple heterogeneous tasks related to eye gaze estimation and semantic user understanding, including eye segmentation, blink detection, emotive expression classification, and pupil and cornea center estimation, using both hand-labeled supervision and model-based supervision, to improve accuracy and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a single deep neural network is trained to perform multiple heterogeneous tasks (eye segmentation, blink detection, expression classification, pupil and cornea center estimation), then measurement precision and reliability are improved, but device complexity increases
Solution Approach 1:
A single deep neural network is designed to perform multiple heterogeneous tasks simultaneously including eye segmentation, blink detection, emotive expression classification, and pupil and cornea center estimation. This multi-functional approach consolidates what would traditionally require separate processing systems into one unified network, improving measurement precision while managing device complexity through integration.
2Ease of operation
If off-axis camera is used for eye imaging in head-mounted devices, then ease of operation and adaptability are improved, but measurement precision deteriorates due to imaging difficulty
Solution Approach 1:
The neural network is trained to compensate for the specific challenges of off-axis imaging by learning from datasets captured with off-axis cameras. The network adapts its parameters to handle the distorted perspectives and lighting conditions inherent in head-mounted device imaging, thereby maintaining measurement precision despite the off-axis configuration.
3Measurement precision
If multiple training tasks are performed sequentially with different supervision methods, then measurement precision is improved, but loss of time increases
Solution Approach 1:
The training process is structured in two sequential phases: first, the feature encoding base network is pre-trained using eye segmentation data with hand-labeled supervision to establish robust feature extraction capabilities. Second, task-specific networks are trained using model-based supervision derived from the pre-trained features. This preliminary action of pre-training the base network accelerates subsequent task-specific training and improves overall measurement precision.
4Reliability
If hand-labeled supervision and model-based supervision are both used for training, then reliability is improved, but device complexity increases
Solution Approach 1:
The feature encoding base network serves as an intermediary that processes input images and generates features used by multiple task-specific networks. This intermediary structure allows hand-labeled supervision (for base network training) and model-based supervision (for task-specific network training) to work together systematically, improving reliability while organizing complexity through a clear hierarchical architecture.
Data Source
AI summary
Techniques related to the computation of gaze vectors of users of wearable devices are disclosed. A neural network may be trained through first and second training steps. The neural network may include a set of feature encoding layers and a plurality of sets of task-specific layers that each operate on an output of the set of feature encoding layers. During the first training step, a first image of a first eye may be provided to the neural network, eye segmentation data may be generated using the neural network, and the set of feature encoding layers may be trained. During the second training step, a second image of a second eye may be provided to the neural network, network output data may be generated using the neural network, and the plurality of sets of task-specific layers may be trained.