Event Camera Gaze Tracking Neural Network
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing gaze tracking systems require high bandwidth and power consumption due to the need for shutter-based camera images, which is inefficient and generates excessive heat.
Innovation Solution
The use of event cameras with neural networks for gaze tracking, where pixel events are processed to derive gaze characteristics, employing multi-stage neural networks and recurrent neural networks to efficiently determine and refine gaze direction with reduced computational resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If shutter-based camera images are used for gaze tracking, then measurement precision is improved, but use of energy and heat generation worsen
Solution Approach 1:
The patent replaces the mechanical shutter-based camera system with an event camera that uses asynchronous pixel sensors. Each pixel independently detects light intensity changes and generates events without mechanical shutters, thereby maintaining measurement precision while dramatically reducing power consumption and heat generation.
Solution Approach 2:
The patent changes the fundamental operating parameter of the camera from continuous frame-based capture to event-based asynchronous capture. This parameter change allows the system to only process pixels that detect meaningful light changes, reducing overall computational load and energy consumption while maintaining gaze tracking accuracy.
2Measurement precision
If high frame rate images are transmitted for gaze tracking, then measurement precision is improved, but loss of energy worsens
Solution Approach 1:
The patent extracts only the essential information needed for gaze tracking by using event-based pixels that generate signals only when light intensity changes exceed a threshold. This extraction approach eliminates redundant data transmission and processing, reducing energy loss while maintaining the precision required for accurate gaze tracking.
3Productivity
If event camera data is processed using neural networks, then productivity is improved, but device complexity increases
Solution Approach 1:
The patent segments the neural network processing into distinct functional components that handle different aspects of event camera data analysis. This segmentation enables more efficient processing of the asynchronous event stream while managing device complexity through modular architecture.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach enables efficient gaze tracking with reduced data and power consumption, maintaining temporal consistency and accuracy even in ambiguous scenarios like occlusions, while providing faster processing and lower energy usage.
Implementation Method 1
Each respective pixel event is generated in response to a respective pixel sensor detecting a change in light intensity of the light at a respective event camera pixel that exceeds a comparator threshold
Data Source
AI summary
One implementation involves a device receiving a stream of pixel events output by an event camera. The device derives an input image by accumulating pixel events for multiple event camera pixels. The device generates a gaze characteristic using the derived input image as input to a neural network trained to determine the gaze characteristic. The neural network is configured in multiple stages. The first stage of the neural network is configured to determine an initial gaze characteristic, e.g., an initial pupil center, using reduced resolution input(s). The second stage of the neural network is configured to determine adjustments to the initial gaze characteristic using location-focused input(s), e.g., using only a small input image centered around the initial pupil center. The determinations at each stage are thus efficiently made using relatively compact neural network configurations. The device tracks a gaze of the eye based on the gaze characteristic.