Multi-Sensor Fusion DNN for Accurate Overlap-Region Object Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing autonomous systems face challenges in accurately detecting and tracking static and dynamic objects due to inconsistent and noisy data from overlapping sensor fields of view, leading to inaccurate object detection and tracking, which can propagate through object tracking, planning, and control operations.
Innovation Solution
A multi-sensor fusion machine learning model, such as a deep neural network (DNN), fuses data from multiple machine learning models, incorporating additional channels like location priors, velocity images, and instance/appearance images to enhance accuracy and precision, especially at sensor overlap regions, through end-to-end training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If hand crafted or rule-based algorithms are used to stitch together outputs from multiple DNNs, then the system can integrate data from multiple sensors, but the fused predictions along borders or overlap regions become inconsistent and inaccurate
Solution Approach 1:
The patent merges multiple DNNs into a single multi-sensor fusion DNN that processes sensor data from multiple sensors simultaneously. Instead of stitching outputs from separate DNNs using rule-based algorithms, the unified model learns to fuse sensor data and generate consistent predictions across all regions including overlap regions, eliminating the boundary inconsistency problem.
Solution Approach 2:
The patent introduces an intermediate representation layer within the multi-sensor fusion DNN that processes and harmonizes features from different sensors before final prediction. This intermediate processing stage acts as a mediator that learns to reconcile differences between sensor fields of view, producing consistent predictions in overlap regions without requiring post-processing stitching algorithms.
2Ease of operation
If individual DNNs are deployed for each sensor, then each sensor can process data independently, but objects near image borders disappear and reappear causing noisy results and reduced tracking accuracy
Solution Approach 1:
The patent combines multiple independent sensor processing streams into a unified multi-sensor fusion DNN that maintains continuous object detection across sensor boundaries. The model processes data from multiple sensors simultaneously and correlates detections across frames, ensuring that objects near borders are tracked continuously without disappearing and reappearing, thereby improving tracking reliability while preserving the independence of individual sensor data streams through the unified architecture.
3Area of stationary object
If multiple DNNs with different fields of view are used, then broader coverage is achieved, but duplicate objects may be detected in overlap regions leading to false determinations
Solution Approach 1:
The patent merges multiple DNNs with different fields of view into a single multi-sensor fusion DNN that processes all sensor inputs simultaneously. The unified model learns to identify and consolidate duplicate detections in overlap regions through its training process, generating a single accurate prediction per object rather than multiple duplicate predictions, thereby maintaining broad coverage while improving detection accuracy.
Solution Approach 2:
The patent implements a loss function that provides feedback during training to penalize duplicate object detections in overlap regions. The model learns from this feedback to correctly identify and merge duplicate detections, adjusting its internal parameters to produce consistent predictions across sensor boundaries. This feedback mechanism enables the model to eliminate false duplicate determinations while preserving broad field of view coverage.
Data Source
AI summary
In various examples, a multi-sensor fusion machine learning model—such as a deep neural network (DNN)—may be deployed to fuse data from a plurality of individual machine learning models. As such, the multi-sensor fusion network may use outputs from a plurality of machine learning models as input to generate a fused output that represents data from fields of view or sensory fields of each of the sensors supplying the machine learning models, while accounting for learned associations between boundary or overlap regions of the various fields of view of the source sensors. In this way, the fused output may be less likely to include duplicate, inaccurate, or noisy data with respect to objects or features in the environment, as the fusion network may be trained to account for multiple instances of a same object appearing in different input representations.