Gaze Prediction Neural Network for Autonomous Vehicle Agent Awareness
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional gaze prediction systems for autonomous vehicles rely on face or head detectors, which have low recall rates when agents are not facing the camera, wearing hats, or looking downward, leading to inaccurate gaze estimation.
Innovation Solution
A gaze prediction neural network that directly predicts gaze direction from raw sensor data, using a combination of camera images and point clouds to generate accurate 2.5D gaze predictions, and generates awareness signals indicating agent awareness, allowing for improved trajectory planning and behavior prediction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If face or head detectors are used for gaze prediction, then the system structure is simple, but the recall rate is low when agents are not facing the camera, wearing hats, or looking downward
Solution Approach 1:
The patent replaces traditional mechanical/algorithmic face and head detectors with a neural network model that directly predicts gaze direction from raw sensor data. This substitution enables the system to detect gaze even when agents are not facing the camera, wearing hats, or looking downward, thereby improving recall rate while maintaining reasonable system complexity through end-to-end learning.
Solution Approach 2:
The patent changes the input parameters from processed face/head detection outputs to raw sensor data (camera images and point clouds). By operating directly on raw data with appropriate preprocessing, the system can detect gaze in more conditions including when agents are wearing hats or looking downward, improving reliability without requiring complex detection pipelines.
2Measurement precision
If 2D camera images are used for gaze estimation, then the system is simple to implement, but the gaze estimation accuracy is insufficient
Solution Approach 1:
The patent merges 2D camera images with 3D point cloud data to create a more comprehensive representation of the agent. This combination allows the neural network to estimate gaze direction more accurately by leveraging both visual appearance information from images and spatial geometric information from point clouds, achieving superior precision compared to using either modality alone.
Solution Approach 2:
The patent transitions from purely 2D image-based gaze estimation to a 2.5D estimation by incorporating 3D point cloud data. This dimensional enhancement provides depth and spatial context, enabling more accurate gaze direction prediction while maintaining computational feasibility through efficient point cloud processing and embedding techniques.
3Adaptability or versatility
If human-programmed logic is used for full-vehicle predictions, then the system is interpretable, but the ability to predict agent awareness and behavior is limited
Solution Approach 1:
The patent replaces human-programmed logic with a neural network model for predicting agent awareness and behavior. This substitution enables the system to capture complex patterns in agent gaze and behavior that are difficult to encode in rules, improving adaptability to diverse driving scenarios while accepting increased model complexity that can be managed through efficient architecture design.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for predicting gaze and awareness using a neural network model. One of the methods includes obtaining sensor data (i) that is captured by one or more sensors of an autonomous vehicle and (ii) that characterizes an agent that is in a vicinity of the autonomous vehicle in an environment at a current time point. The sensor data is processed using a gaze prediction neural network to generate a gaze prediction that predicts a gaze of the agent at the current time point. The gaze prediction neural network includes an embedding subnetwork that is configured to process the sensor data to generate an embedding characterizing the agent, and a gaze subnetwork that is configured to process the embedding to generate the gaze prediction.