Multi-Modal Sensor Fusion with Latent Trajectory Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multi-modal sensor fusion systems face challenges in integrating data from different sensors due to varying coordinate systems, incompatible detection rates, and noise, which are critical for real-time vehicle safety applications.
Innovation Solution
A method for multi-modal sensor fusion that aligns and matches spatiotemporal trajectories from camera and RADAR/LIDAR sensors using a latent state model and optimization algorithms to improve robustness and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multi-modal sensor fusion is performed using traditional methods (Hungarian algorithm with Euclidean distance), then target matching can be achieved, but measurement precision and reliability deteriorate due to varying coordinate systems, incompatible detection rates, and noise
Solution Approach 1:
The patent introduces a latent state model as an intermediary that bridges camera and RADAR/LIDAR trajectories. Instead of directly matching trajectories in their native coordinate systems, the model maps both to a latent state space where correspondence can be established. This mediator resolves the coordinate system incompatibility and detection rate mismatch problems by providing a common representation framework.
Solution Approach 2:
The patent transforms the matching problem by changing the parameter space. Rather than using raw coordinate parameters and Euclidean distance, the system converts trajectories into latent state representations with optimized parameters. The objective function uses transformed parameters (log-likelihood ratios, normalized distances) that are more suitable for cross-modal comparison, improving measurement precision.
2Speed
If real-time processing is required for vehicle safety applications, then processing speed must be high, but measurement precision deteriorates due to noise and incompatible detection rates from different sensors
Solution Approach 1:
The patent performs preliminary actions by pre-processing trajectories into latent state representations before matching. The system pre-computes trajectory features, normalizes detection rates, and establishes coordinate transformations in advance. This preliminary preparation reduces the computational burden during real-time matching, allowing high processing speed without sacrificing precision.
Solution Approach 2:
The patent implements feedback mechanisms through the latent state model that continuously refines trajectory matching. The model uses probabilistic feedback from both camera and RADAR/LIDAR observations to update latent states, improving detection accuracy over time. This feedback loop allows the system to maintain high precision even with noisy inputs while operating in real-time.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A system for computing correspondence between multiple sensor observations, comprising: an interface for receiving first spatiotemporal datasets and second spatiotemporal datasets of a plurality of moving objects, based on signals originating from a first sensor and a second sensor capturing different types of signal; processing circuitry, configured to: generating a plurality of first spatiotemporal object trajectories and a plurality of second spatiotemporal object trajectories based on the first spatiotemporal datasets and the second spatiotemporal datasets respectively; computing a distance metric between the first and second spatiotemporal object trajectories, by a respective mapping from a respective domain into a predetermined domain; executing a matching computation between the mappings of the plurality of first and second spatiotemporal object trajectories by employing a distance metric in order to calculate a similarity between each pair of mappings of first and second spatiotemporal object trajectories, and outputting an outcome of the matching computation.