Multi-Camera Object Fusion Tracking With Low-Latency Smoothing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multi-camera fusion approaches in autonomous systems introduce latency in object tracking due to the need to process multiple frames of camera input, leading to delays in reaction time and potential errors in downstream operations.
Innovation Solution
A low-latency fusion and tracking technique that identifies subsets of input objects from multiple sensors based on similarity criteria, using a cost optimization algorithm to associate objects across cameras and perform smoothing operations to generate output objects with reduced latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple frames of camera input are processed by the DNN to reduce detection errors, then reliability of object detection is improved, but latency of the perception system increases
Solution Approach 1:
The patent segments the object tracking process into two independent levels: (1) DNN-based object detection that processes single frames to generate detected objects, and (2) multi-camera fusion tracking that processes detection results across multiple cameras. This segmentation allows the DNN to operate on single frames (reducing latency) while the fusion tracker provides multi-frame temporal filtering (maintaining reliability). The tracker receives detected objects from multiple cameras and performs association and filtering independently, without requiring re-processing through the DNN.
Solution Approach 2:
The patent introduces an intermediary multi-camera fusion tracker that sits between the DNN detection pipeline and downstream applications. This intermediary component receives detection results from multiple cameras, performs temporal filtering and association across frames, and outputs refined tracking results. It acts as a mediator that provides the temporal filtering benefits of multi-frame processing without requiring the expensive DNN to process multiple frames, thus reducing latency while maintaining reliability.
2Measurement precision
If high-level fusion approaches perform additional filtering in the tracker, then object tracking accuracy is improved, but latency in producing object characteristics increases
Solution Approach 1:
The patent segments filtering operations into two stages: (1) DNN-based filtering within individual camera perception pipelines that processes single frames, and (2) lightweight temporal filtering in the multi-camera fusion tracker that operates on detection results. This segmentation allows accurate tracking through multi-camera association while avoiding the latency penalty of running additional DNN filtering passes on multiple frames.
Solution Approach 2:
The patent changes the operational parameters of the tracking system by operating the fusion tracker on detection results rather than on raw images. This parameter change allows the tracker to perform association and filtering operations on already-processed object detections, significantly reducing computational latency while maintaining tracking accuracy through multi-camera data fusion and temporal consistency checks.
3Reliability
If the system waits for multiple frames to be processed by the DNN, then false positive detections are reduced, but reaction time of the perception system increases
Solution Approach 1:
The patent segments the detection and tracking functions so that the DNN processes single frames for rapid detection, while a separate multi-camera fusion tracker performs temporal filtering to reduce false positives. This segmentation enables the system to achieve the false positive reduction benefit of multi-frame processing without requiring the slow DNN to process multiple frames sequentially, thus improving reaction time while maintaining detection reliability.
Solution Approach 2:
The patent substitutes the mechanical DNN processing system with a lighter-weight fusion tracker for temporal filtering operations. Instead of using the computationally intensive DNN to process multiple frames for false positive reduction, the system uses the fusion tracker to perform association and temporal consistency checks on detection results, achieving the same reliability improvement with much lower computational overhead and faster reaction time.
Data Source
AI summary
In various examples, a technique for low-latency fusion and tracking of objects from multiple cameras is disclosed that includes determining a plurality of input objects and corresponding object characteristics, individual input objects and respective object characteristics being determined based at least on an image generated using a respective camera. The technique also includes identifying at least one subset of the plurality of input objects, the at least one subset corresponding to a respective physical object and comprising at least one input object that satisfies a similarity criterion. The technique further includes generating an output object associated with one or more smoothed object characteristics, the output object being generated based at least on two or more input objects included in the at least one subset of the plurality of input objects. The at least one subset corresponds to a physical object that is visible to two or more cameras.