Multi-Sensor Fusion DNN for Consistent Overlap-Region Predictions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing autonomous driving systems face challenges in accurately detecting static and dynamic objects, obstacles, and features across overlapping fields of view from different sensors, leading to noisy and inconsistent data.
Innovation Solution
A multi-sensor fusion machine learning model, such as a deep neural network (DNN), is used to fuse data from multiple individual machine learning models, accounting for learned associations between boundary or overlap regions of different sensor fields of view.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If hand crafted or rule-based algorithms are used to stitch together outputs from multiple DNNs, then the system can process data from multiple sensors, but the fused predictions in overlap regions become inconsistent and inaccurate
Solution Approach 1:
The patent merges multiple DNNs into a unified multi-sensor fusion DNN that processes data from multiple sensors simultaneously. Instead of separately running individual DNNs and stitching their outputs with rule-based algorithms, the fused DNN integrates sensor inputs at the feature level and produces unified predictions, eliminating inconsistencies in overlap regions while maintaining multi-sensor processing capability
Solution Approach 2:
The patent introduces an intermediary fusion layer within the DNN architecture that mediates between individual sensor inputs and final predictions. This intermediary layer learns to reconcile conflicting predictions from different sensors in overlap regions through learned association models, rather than using rigid rule-based stitching algorithms
2Ease of operation
If individual DNNs are deployed for each sensor, then each sensor can be processed independently, but the fused output contains duplicate and noisy data in boundary regions
Solution Approach 1:
The patent combines independent sensor processing with unified object detection by merging multiple DNNs into a single multi-sensor fusion DNN. This unified model processes each sensor's data independently through separate input branches but integrates their features internally, producing a single consistent set of object predictions that eliminates duplicates in boundary regions while preserving the benefits of independent sensor processing
Solution Approach 2:
The patent transitions from processing sensors in separate 2D image spaces to a unified 3D spatial representation. By projecting sensor data into a common 3D space and performing fusion in this additional dimension, the system can identify and merge duplicate object detections across sensor boundaries, reducing noise while maintaining independent processing capabilities
Data Source
AI summary
In various examples, a multi-sensor fusion machine learning model—such as a deep neural network (DNN)—may be deployed to fuse data from a plurality of individual machine learning models. As such, the multi-sensor fusion network may use outputs from a plurality of machine learning models as input to generate a fused output that represents data from fields of view or sensory fields of each of the sensors supplying the machine learning models, while accounting for learned associations between boundary or overlap regions of the various fields of view of the source sensors. In this way, the fused output may be less likely to include duplicate, inaccurate, or noisy data with respect to objects or features in the environment, as the fusion network may be trained to account for multiple instances of a same object appearing in different input representations.