Heterogeneous Sensor Fusion Merging Network for Object Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional non-maximal suppression methods for integrating object detection information from multiple camera images in autonomous vehicles face challenges in determining the optimal threshold for merging bounding boxes, leading to either incorrect merging or missed integrations, and require constant updates, resulting in inefficient computation.
Innovation Solution
A learning method using a Deep Neural Network (DNN) with a concatenating, determining, and loss unit to generate pair feature, determination, and box regression vectors, applying cross-entropy and smooth-L1 methods to integrate object detection information from multiple camera images without additional computation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a low threshold is used for non-maximal suppression, then more bounding boxes are merged, but unrelated bounding boxes may be incorrectly merged
Solution Approach 1:
The patent introduces an intermediary classification step between threshold-based filtering and bounding box merging. A trained model classifies candidate merged bounding boxes to determine whether they represent valid objects or false merges, enabling accurate merging without requiring manual threshold tuning.
Solution Approach 2:
The patent replaces the mechanical threshold-based filtering system with a learned classification system. Instead of using fixed or manually tuned overlap thresholds, the system uses a trained neural network to intelligently determine which bounding boxes should be merged, substituting rigid mechanical rules with adaptive learned behavior.
2Reliability
If a high threshold is used for non-maximal suppression, then merging accuracy is improved, but related bounding boxes may not be integrated
Solution Approach 1:
The classification model acts as an intermediary that processes all candidate merged bounding boxes regardless of their overlap ratio. This eliminates the need for high thresholds that would filter out potentially valid merges, as the learned classifier can accurately identify true objects even with low overlap ratios.
Solution Approach 2:
The patent changes the parameter from a fixed threshold value to a dynamic classification probability. Instead of filtering based on a static overlap ratio parameter, the system uses the output probability from the trained classification model to determine merging, allowing flexible adaptation to different scenarios without manual parameter adjustment.
3Reliability
If manual threshold updates are performed to adapt to different scenarios, then merging accuracy is maintained, but computational overhead and complexity increase
Solution Approach 1:
The patent performs preliminary action by training the classification model offline before deployment. The model learns optimal merging criteria from training data during a separate preparation phase, eliminating the need for manual threshold updates during runtime operations. This shifts complexity from operational complexity to training complexity.
Solution Approach 2:
The system achieves self-service through the trained classification model that automatically determines merging decisions without human intervention. The model adapts to different scenarios through its training data rather than requiring manual threshold adjustments, making the system self-sufficient and reducing operational complexity.
4Loss of information
If object detection is performed on the integrated image, then complete object information is obtained, but redundant computation occurs
Solution Approach 1:
The patent extracts only the necessary object detection results from individual camera images before integration. Instead of performing full object detection on the computationally expensive integrated image, the system extracts detected objects and their bounding boxes from each camera's processing results and directly integrates these extracted elements.
Solution Approach 2:
The patent merges detection results from multiple cameras at the bounding box level rather than re-processing integrated images. By combining the already-processed detection outputs from individual cameras with the classification model, the system avoids redundant computation while maintaining complete object information.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A learning method for generating integrated object detection information of an integrated image by integrating first object detection information and second object detection information is provided. The method includes steps of: (a) a learning device, if the first object detection information and the second object detection information is acquired, instructing a concatenating network included in a DNN to generate pair feature vectors including information on pairs of first original ROIs and second original ROIs; (b) the learning device instructing a determining network included in the DNN to apply FC operations to the pair feature vectors, to thereby generate (i) determination vectors and (ii) box regression vectors; (c) the learning device instructing a loss unit to generate an integrated loss, and performing backpropagation processes by using the integrated loss, to thereby learn at least part of parameters included in the DNN.