Driver Attention Heat Maps for Semantic Segmentation Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current semantic segmentation network models for autonomous driving do not account for the varying importance of pixels, treating all pixels equally during training, which is not reflective of human attention and relevance.
Innovation Solution
A system and method that incorporates pixel weighting based on driver eye movement tracking, using outside view and driver-facing cameras to create a pixel weighted heat map, which is then used to train semantic and attention neural network models to prioritize relevant pixels for improved autonomous driving functionality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all pixels are treated equally during training of segmentation network models, then the training process is simple and uniform, but the model fails to capture the varying importance of different pixels, reducing segmentation accuracy
Solution Approach 1:
The patent applies local quality by assigning different weights to different pixels based on their importance, rather than treating all pixels uniformly. The pixel weighting mechanism assigns higher weights to pixels corresponding to regions where the driver is looking (determined by eye tracking), and lower weights to other regions. This allows the model to focus computational resources on important regions while maintaining simplicity in the overall training framework.
Solution Approach 2:
The patent changes the parameter of pixel importance by introducing a weighting factor that modulates the contribution of each pixel to the loss function during training. This parameter change transforms the uniform pixel treatment into a differentiated approach where pixels are weighted according to their relevance, improving segmentation accuracy without fundamentally changing the training architecture.
2Measurement precision
If pixel weighting based on driver attention is implemented, then the model can prioritize relevant pixels for better segmentation, but additional hardware (eye tracking camera) and processing steps are required
Solution Approach 1:
The patent uses an intermediary approach by introducing a driver-facing camera that captures eye movement data as an intermediate input. This intermediary data source provides attention information that is then integrated with the external camera images through image registration and pixel weighting. The intermediary eye tracking data bridges the gap between driver cognitive state and model processing, enabling attention-based weighting without requiring direct modification of the core segmentation architecture.
3Adaptability or versatility
If driver eye movement data is integrated with external camera images, then pixel importance can be determined, but image calibration and alignment processes are required
Solution Approach 1:
The patent applies preliminary action by performing image calibration and alignment before the main segmentation process. The eye movement data and external camera images are pre-registered and aligned in advance, creating a calibrated coordinate system that maps driver attention directly to image pixels. This preliminary calibration step, though adding processing complexity, enables the subsequent pixel weighting to be performed efficiently without requiring complex real-time alignment during segmentation.
Data Source
AI summary
Methods of creating trained semantic segmentation network models and operating vehicles using the model. One example method includes an outside view camera configured to capture images that represent an artificial representation of the driver's view, a driver-facing camera configured to capture a driver's eye movements, and an electronic controller. The electronic controller is configured to receive images from the cameras; calibrate the image of the driver's eye movement with the artificial driver view; create a pixel weighted heat map of the calibrated images; create a trained semantic segmentation neural network model and a trained attention neural network model using the pixel weighted heat map and the artificial driver view; and operate the vehicle using the trained semantic segmentation neural network model and the trained attention neural network model.