Sequential DNN Motion Prediction From Images for Autonomous Driving
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional computer vision algorithms for autonomous driving are sensitive to non-rigid motion and viewing direction, fail to account for object context, and rely on fixed time baselines, leading to inferior temporal information predictions, especially for pedestrians.
Innovation Solution
A sequential deep neural network (DNN) is trained using cross-sensor fusion to predict time-to-collision, 2D, and 3D object motion from image data alone, employing stateless or stateful methods for improved accuracy and efficiency, with automatic ground truth generation and temporal augmentation to enhance robustness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional computer vision algorithms are used to estimate local motion information, then the system can extract motion data, but the predictions are sensitive to non-rigid motion and viewing direction, leading to inferior prediction accuracy
Solution Approach 1:
The patent replaces conventional computer vision algorithms with a deep neural network-based system that uses optical flow fields and sensor fusion. This substitution enables the system to handle non-rigid motion and varying viewing directions more robustly by learning temporal patterns from multi-sensor data, thereby improving prediction accuracy while reducing sensitivity to motion complexity.
Solution Approach 2:
The patent employs sensor fusion that combines data from multiple sources including cameras, LIDAR, and IMU to create a composite temporal information prediction system. This composite approach leverages the strengths of each sensor type to compensate for individual weaknesses, particularly in handling non-rigid motion and different viewing directions, thus improving overall reliability and accuracy.
2Loss of information
If conventional computer vision algorithms are used, then the system can process visual data, but it fails to account for object context, resulting in less informative predictions
Solution Approach 1:
The patent merges visual data from cameras with contextual information from LIDAR and sensor data to create a comprehensive temporal prediction model. This combination ensures that object context is preserved and utilized in predictions, reducing information loss and improving the informativeness of temporal estimates.
Solution Approach 2:
The patent transitions from purely 2D image-based computer vision to a multi-dimensional approach by incorporating 3D spatial data from LIDAR and temporal data from inertial sensors. This dimensional expansion allows the system to capture object context more effectively, preserving information that would be lost in traditional 2D visual processing alone.
3Adaptability or versatility
If fixed time baselines are used to compute temporal information, then the computation is simple, but the system cannot separately account for different objects, scenarios, and conditions, resulting in less accurate predictions
Solution Approach 1:
The patent implements dynamic time baseline selection where the temporal computation window adapts based on object type, motion characteristics, and scenario context. Instead of using fixed time baselines, the system dynamically adjusts the temporal scope for different objects and conditions, enabling versatile adaptation while managing complexity through learned temporal models.
Solution Approach 2:
The patent changes the temporal parameters of the computation based on the specific object and scenario being analyzed. By adjusting time baseline length, sampling frequency, and integration window according to object characteristics and environmental conditions, the system achieves high adaptability across different situations while using efficient computational methods tailored to each parameter set.
Data Source
AI summary
In various examples, a sequential deep neural network (DNN) may be trained using ground truth data generated by correlating (e.g., by cross-sensor fusion) sensor data with image data representative of a sequences of images. In deployment, the sequential DNN may leverage the sensor correlation to compute various predictions using image data alone. The predictions may include velocities, in world space, of objects in fields of view of an ego-vehicle, current and future locations of the objects in image space, and/or a time-to-collision (TTC) between the objects and the ego-vehicle. These predictions may be used as part of a perception system for understanding and reacting to a current physical environment of the ego-vehicle.