Sequential DNN Prediction of Time-to-Collision From Image Sequences
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional computer vision algorithms for autonomous driving are sensitive to non-rigid motion and viewing direction, fail to account for object context, and rely on fixed time baselines, leading to inferior predictions of temporal information such as time-to-collision and object motion.
Innovation Solution
A sequential deep neural network (DNN) is trained using correlated sensor and image data to predict time-to-collision, 2D, and 3D object motion without requiring sensor data during deployment, employing cross-sensor fusion, automatic ground truth generation, and stateless or stateful training methods for improved accuracy and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional computer vision algorithms are used to estimate local motion information, then the system can extract motion data, but the predictions are sensitive to non-rigid motion and viewing direction, leading to inferior prediction accuracy
Solution Approach 1:
The patent replaces conventional computer vision algorithms with a deep neural network model that processes image sequences to predict temporal information. The DNN learns robust features from training data that includes various viewing directions and non-rigid motions, making the system insensitive to these variations during deployment. This substitution of mechanical/computer vision algorithms with a learned deep learning model resolves the sensitivity issue.
Solution Approach 2:
The system performs preliminary training of the deep neural network using extensive training data that encompasses various non-rigid motions and viewing directions. This pre-training phase allows the model to learn invariant features before actual deployment, making it robust to these variations without requiring complex runtime adjustments or sensor inputs.
2Adaptability or versatility
If conventional algorithms with fixed time baselines are used, then the system can compute temporal information, but it cannot separately account for different objects, scenarios, and conditions, resulting in less informative predictions
Solution Approach 1:
The patent implements a dynamic approach where the deep neural network adapts its predictions based on the specific object, scenario, and conditions present in the input image sequences. The model processes contextual information from multiple frames and dynamically adjusts its predictions for different object types and situations, rather than using a fixed baseline approach. This enables the system to account for different objects, scenarios, and conditions separately and accurately.
3Measurement precision
If sensor data is required as input during deployment, then the system can generate ground truth data for training, but it requires dense optical flow or motion of the vehicle during deployment, reducing ease of operation
Solution Approach 1:
The patent creates a copy of the ground truth generation capability within the deep neural network itself. During training, the model learns to predict temporal information that would normally require sensor data and vehicle motion. Once trained, the model can generate accurate predictions using only image sequences, effectively copying the ground truth generation capability into the model without requiring actual sensors or vehicle motion during deployment.
4Ease of operation
If sequential deep neural networks are used to predict temporal information from images alone, then the system can operate without sensor data inputs, but it requires sophisticated training methods like cross-sensor fusion and temporal augmentation to achieve accurate results
Solution Approach 1:
The patent performs all complex training operations in advance during the model training phase. Cross-sensor fusion, temporal augmentation, and ground truth generation are all accomplished during training to create a robust model. Once trained, the model operates simply with only image data inputs, as the complexity has been resolved in the preliminary training stage. This separates the complexity of model creation from the simplicity of model deployment.
Data Source
AI summary
In various examples, a sequential deep neural network (DNN) may be trained using ground truth data generated by correlating (e.g., by cross-sensor fusion) sensor data with image data representative of a sequences of images. In deployment, the sequential DNN may leverage the sensor correlation to compute various predictions using image data alone. The predictions may include velocities, in world space, of objects in fields of view of an ego-vehicle, current and future locations of the objects in image space, and/or a time-to-collision (TTC) between the objects and the ego-vehicle. These predictions may be used as part of a perception system for understanding and reacting to a current physical environment of the ego-vehicle.