Sequential DNN Temporal Prediction for Image-Only Object Motion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional computer vision algorithms for autonomous driving are sensitive to non-rigid motion and viewing direction, fail to account for object context, and rely on fixed time baselines, leading to inferior temporal information predictions, especially for pedestrians.
Innovation Solution
A sequential deep neural network (DNN) is used to predict time-to-collision, 2D, and 3D object motion by correlating sensor data with image data, leveraging cross-sensor fusion, automatic ground truth data generation, and stateless or stateful training methods for improved accuracy and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional computer vision algorithms are used to estimate local motion information, then the system can extract motion data, but the predictions are sensitive to non-rigid motion and viewing direction, leading to inferior prediction accuracy
Solution Approach 1:
The patent replaces conventional computer vision algorithms with a deep neural network model that processes image sequences to predict temporal information. The DNN learns robust features from training data that includes diverse viewing directions and non-rigid motions, making the system insensitive to these variations during deployment.
Solution Approach 2:
The system performs preliminary training of the deep neural network using extensive training data that covers various non-rigid motions and viewing directions. This pre-training ensures the model has already learned to handle these variations before actual deployment, improving both accuracy and reliability without requiring sensor data during operation.
2Loss of information
If conventional computer vision algorithms are used, then the system can process images, but it fails to account for object context, resulting in inability to capture how objects move relative to static surroundings
Solution Approach 1:
The patent introduces a new dimension of context understanding by processing sequences of images through a deep neural network that learns temporal and spatial relationships. The DNN captures object context by analyzing how objects move relative to static surroundings across multiple frames, preserving motion information that conventional algorithms miss.
3Device complexity
If conventional systems use fixed time baselines to compute temporal information, then the computation is simple, but the predictions are less informative and accurate because they do not separately account for different objects, scenarios, and conditions
Solution Approach 1:
The patent implements a dynamic computation mechanism where the deep neural network adapts its processing based on the input image sequences. The model learns different temporal baselines and processing strategies for different objects, scenarios, and conditions during training, allowing it to handle diverse situations with varying complexity requirements.
Solution Approach 2:
The system changes computational parameters dynamically by using a trained DNN that adjusts its internal weights and processing methods based on the specific input scenario. This allows the system to optimize prediction accuracy for different objects and conditions without requiring manual configuration of fixed time baselines.
4Measurement precision
If sensor data is required as input during deployment to generate accurate predictions, then the prediction accuracy can be maintained, but the system cannot generate predictions from images alone
Solution Approach 1:
The patent implements a self-service system where the deep neural network is trained offline using correlated sensor data and image data to learn the relationship between visual features and temporal information. During deployment, the trained model independently generates accurate predictions from image sequences alone, making the system self-sufficient and adaptable to various deployment scenarios.
Data Source
AI summary
In various examples, a sequential deep neural network (DNN) may be trained using ground truth data generated by correlating (e.g., by cross-sensor fusion) sensor data with image data representative of a sequences of images. In deployment, the sequential DNN may leverage the sensor correlation to compute various predictions using image data alone. The predictions may include velocities, in world space, of objects in fields of view of an ego-vehicle, current and future locations of the objects in image space, and/or a time-to-collision (TTC) between the objects and the ego-vehicle. These predictions may be used as part of a perception system for understanding and reacting to a current physical environment of the ego-vehicle.