Multi-Modal Fusion with Projection Alignment for sUAS Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multi-modal fusion techniques face challenges with spatial asynchrony and limited interpretability, particularly in detecting and tracking small unmanned aerial systems (sUAS) due to spatial misalignment and inefficient information sharing across different data sources.
Innovation Solution
A novel Multi-Modal Region Proposal Network (MMRPN) with a projection function aligns and synergistically shares information across modalities, using Faster RCNN and Feature Pyramid Networks (FPN) to enhance detection and tracking of sUAS by optimizing information sharing and reducing latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing multi-modal fusion techniques are used, then integration of multi-modal data can be achieved, but spatial misalignment and spatial asynchrony occur leading to reduced detection accuracy
Solution Approach 1:
The patent introduces an intermediary projection function that maps proposals from one modality to another modality's coordinate system. This mediator enables spatial alignment between asynchronous multi-modal data by converting proposals between different spatial representations, thereby resolving the spatial misalignment issue while maintaining detection accuracy
Solution Approach 2:
The patent adds a temporal dimension to the fusion process by processing sequences of proposals over time. By considering temporal sequences and using recurrent neural networks, the system handles spatial asynchrony across time, transforming the problem from purely spatial alignment to spatio-temporal alignment in an additional dimension
2Loss of information
If multi-modal data integration is performed, then understanding of situation or activity is improved, but information loss occurs during fusion process
Solution Approach 1:
The patent segments the fusion process into distinct stages: individual modality processing, proposal generation, projection to common coordinate system, and sequential fusion. This segmentation allows information from each modality to be processed and preserved independently before integration, reducing information loss while managing system complexity through modular architecture
Solution Approach 2:
The patent uses region proposal networks that generate multiple candidate proposals (excessive action) rather than directly fusing all data. This partial processing approach allows selective refinement of promising regions while discarding irrelevant information, thereby retaining critical information without requiring complete processing of all multi-modal data
3Productivity
If region proposal networks are used for each mode, then detection proposals can be generated, but computational latency increases
Solution Approach 1:
The patent applies region proposal networks to individual modalities separately before fusion, generating preliminary proposals in advance. This preliminary action allows each modality to independently identify candidate regions beforehand, reducing the computational burden during the fusion stage and thereby decreasing overall processing latency while maintaining detection speed
Data Source
AI summary
A computer-implemented method, and corresponding system, predicts existence of a feature in a scene within an environment for each of at least two modes of observation, e.g., thermal, video color, or multispectral imaging, of the scene. Predicting the existence includes calculating a proposal of the existence of the feature within a respective bounded region of a representation of the scene for each of the at least two modes of each observation of a sequence of observations. The method uses a projection function that synergistically shares information between or among the at least two modes to generate a corresponding fused predicted existence of the feature for each of the at least two modes of each observation, thereby maximizing the benefits of multi-modality. The feature may be an object of interest that may be occluded and moving within the environment. The predicted existence enables the object to be detected, tracked, and/or identified.