Eye-Gaze Pre-Training for Disparity Networks in Object Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing vehicular vision systems face challenges in efficiently training neural networks for object detection due to the labor-intensive manual labeling required for gaze prediction and the mismatch between sparse depth information and continuous gaze data, which hinders effective driver monitoring and autonomous vehicle operations.
Innovation Solution
A pre-training phase using eye-gaze data to automatically generate labels, followed by a supervised training phase that transforms depth information into normalized disparity maps, aligns optical and distance sensor data, and minimizes deviation using L1 loss functions to develop a disparity estimation neural network for object detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual labeling is used for gaze prediction training, then training data accuracy is improved, but training time and labor cost increase significantly
Solution Approach 1:
The system performs preliminary action by using automatically generated gaze labels from eye-tracking data to pre-train the neural network before final supervised training. This preliminary labeling step using automated eye-gaze detection systems reduces the need for extensive manual annotation while providing sufficient initial training data to guide the network's learning process effectively
Solution Approach 2:
The system applies self-service by enabling the neural network to learn from automatically generated gaze predictions that are derived from the same eye-tracking data used during operation. The network essentially trains itself using the inherent gaze information already captured by the sensor system, eliminating the need for separate manual labeling processes
2Device complexity
If sparse depth information is used directly for training, then data processing complexity is reduced, but neural network performance deteriorates due to data mismatch
Solution Approach 1:
The system transforms the depth information parameter by converting sparse depth measurements into normalized disparity maps. This parameter transformation creates a continuous representation from discrete depth points, improving the compatibility between depth data and gaze data while maintaining computational efficiency. The normalization process scales the disparity values to a standard range, enabling effective gradient-based optimization during training
Solution Approach 2:
The system introduces an intermediary transformation process that converts sparse depth information into a format compatible with continuous gaze data. The depth information serves as an intermediary that is processed through normalization and disparity map generation, creating a bridge between the two data types that enables effective joint training without direct mismatch
3Measurement precision
If continuous gaze data is used without transformation, then gaze prediction accuracy is maintained, but compatibility with sparse depth information decreases
Solution Approach 1:
The system applies parameter changes by transforming the continuous gaze data representation to be compatible with sparse depth information. Instead of converting depth to gaze coordinates, the approach transforms the problem by creating normalized disparity maps that can be jointly optimized with gaze predictions, enabling both data types to contribute effectively to the same loss function
Solution Approach 2:
The system addresses the dimensionality mismatch by introducing a new representation dimension through disparity maps. Rather than trying to align gaze (angular) data directly with depth (distance) data, the system creates an intermediate disparity representation that bridges the dimensional gap, allowing both continuous gaze information and sparse depth measurements to be processed in a unified framework
Data Source
AI summary
A method of training a disparity estimation network. The method includes obtaining an eye-gaze dataset having first images with at least one gaze direction associated with each of the first images. A gaze prediction neural network is trained based on the eye-gaze dataset to develop a model trained to provide a gaze prediction for an external image. A depth database is obtained that includes second images having depth information associated with each of the second images. A disparity estimation neural network for object detection is trained based on an output from the gaze prediction neural network and an output from the depth database.