Modular Neural Gaze Estimation From Spatiotemporal Eye Features
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing gaze estimation technologies face challenges in accurately determining the direction of user attention and distance information from images, particularly in complex scenarios like augmented reality and human-computer interaction, due to limitations in feature extraction and temporal relationship analysis.
Innovation Solution
A method and device utilizing artificial intelligence and machine learning techniques, including convolutional neural networks and transformer networks, to obtain target feature maps from images, incorporating attention information, temporal relationships, and pixel distances, to enhance gaze estimation accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional feature extraction methods are used for gaze estimation, then the device complexity is low, but the measurement precision of gaze direction is insufficient
Solution Approach 1:
The patent segments the feature extraction process into multiple specialized modules: a first neural network module extracts spatial features from eye images, a second neural network module extracts temporal features from image sequences, and a third neural network module integrates these features to determine gaze direction. This segmentation allows each module to specialize in specific feature types, improving overall measurement precision while managing system complexity through modular design.
Solution Approach 2:
The patent transitions from traditional 2D image analysis to 4D spatiotemporal feature analysis by incorporating both spatial dimensions (x, y coordinates in the image) and temporal dimension (sequence of frames). The neural networks process features across multiple dimensions, extracting spatial relationships within individual frames and temporal relationships across frame sequences, thereby achieving more accurate gaze estimation.
2Measurement precision
If temporal relationship analysis is incorporated into gaze estimation, then the measurement precision improves, but the processing time increases
Solution Approach 1:
The patent performs preliminary action by pre-processing images to extract key features (eye contours, pupil positions, facial landmarks) before temporal analysis. The first neural network module pre-extracts spatial features from each individual frame, creating a compact feature representation that reduces the computational burden of subsequent temporal analysis by the second neural network module, thereby reducing overall processing time.
Solution Approach 2:
The patent extracts only the most relevant temporal relationships for gaze estimation rather than analyzing all possible temporal variations. The second neural network module selectively extracts temporal features that are most indicative of gaze direction changes, discarding redundant information. This selective extraction maintains measurement precision while minimizing processing time by focusing computational resources on critical temporal patterns.
3Measurement precision
If multiple neural network modules are used for feature extraction, then the measurement precision of gaze estimation improves, but the device complexity increases
Solution Approach 1:
The patent divides the complex neural network system into three distinct, functionally-specialized modules: (1) a first neural network for spatial feature extraction from individual eye images, (2) a second neural network for temporal feature extraction from image sequences, and (3) a third neural network for integrating features and determining gaze direction. Each module has a specific function, making the overall complex system manageable through clear segmentation and reducing the complexity burden on any single component.
Solution Approach 2:
The patent introduces intermediate feature representations as mediators between the input images and the final gaze direction output. The first neural network produces intermediate spatial feature maps, which are then processed by the second neural network to generate intermediate temporal features. These intermediate representations serve as bridges that simplify the computation by breaking down the complex transformation from raw images to gaze direction into manageable stages, each handled by a specialized neural network module.
Data Source
AI summary
An electronic device and method with gaze estimating are disclosed. The method includes obtaining target information of an image, the image including an eye, obtaining a target feature map representing information on the eye in the image based on the target information, and estimating a gaze for the eye in the image based on the target feature map. The target information includes either attention information on the image, or a distance between pixels in the image, or both.