3D Gaze Prediction Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing gaze detection systems struggle with accurate 3D gaze tracking and are limited by camera and screen configuration, requiring re-training for changes in image resolution, focal length, or screen size, which hampers support for stereoscopic displays and 3D applications.
Innovation Solution
A deep learning system that trains a neural network using diverse training images of user eyes with gaze points within and outside the camera plane, allowing it to predict 3D gaze independently of camera and screen configurations by estimating eye position and distance correction, and mapping 2D gaze vectors to 3D gaze directions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a neural network is trained for a specific camera and screen configuration, then 2D gaze prediction accuracy is improved, but the system requires re-training when configuration changes occur
Solution Approach 1:
The patent transforms fixed camera and screen configuration parameters into variable input features that the neural network can process dynamically. By representing focal length, distance to screen, image resolution, and screen size as input parameters rather than fixed training conditions, the system adapts to configuration changes without requiring re-training, while maintaining 2D gaze prediction accuracy.
2Adaptability or versatility
If existing techniques are expanded to 3D gaze detection, then support for stereoscopic displays is improved, but prediction accuracy deteriorates
Solution Approach 1:
The patent segments 3D gaze detection into two independent components: 2D gaze direction detection (using the trained neural network) and depth estimation (using separate depth sensors or techniques). This segmentation allows the system to leverage the accurate 2D detection capability while adding 3D functionality through complementary depth information, thereby maintaining overall prediction accuracy while enabling stereoscopic display support.
3Adaptability or versatility
If diverse training images with gaze points outside camera plane are used, then 3D gaze detection capability is improved, but training complexity increases
Solution Approach 1:
The patent performs preliminary action by pre-processing training images to annotate not only 2D gaze points but also depth information and 3D gaze directions. By preparing this comprehensive 3D training data in advance with proper labeling and organization, the system enables 3D gaze detection capability while managing training complexity through structured data preparation and efficient training pipelines.
Data Source
AI summary
Techniques for generating 3D gaze predictions based on a deep learning system are described. In an example, the deep learning system includes a neural network. The neural network is trained with training images generated by cameras and showing eyes of user while gazing at stimulus points. Some of the stimulus points are in the planes of the camera. Remaining stimulus points are not un the planes of the cameras. The training includes inputting a first training image associated with a stimulus point in a camera plane and inputting a second training image associated with a stimulus point outside the camera plane. The training minimizes a loss function of the neural network based on a distance between at least one of the stimulus points and a gaze line.