3D Object Velocity Fusion Using Radar and Optical Flow
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for estimating object velocities using camera or radar sensors alone are limited by the sparsity and unreliability of radar returns and the failure of optical flow techniques under dynamic conditions, leading to inaccurate and unreliable velocity estimations.
Innovation Solution
A multi-sensor fusion technique that combines sparse radar returns with optical flow features, utilizing k-means clustering and deformable cross-attention to correct errors, and incorporates temporal consistency checks for robust velocity estimation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If radar sensor alone is used for velocity estimation, then device complexity is reduced, but measurement precision deteriorates due to sparsity and unreliability of radar returns
Solution Approach 1:
The patent combines radar sensor data with camera optical flow data to create a fused velocity estimation system. The radar provides reliable radial velocity measurements while the camera provides dense optical flow fields, and their fusion through attention mechanisms achieves superior velocity estimation accuracy compared to either sensor alone.
Solution Approach 2:
The system uses a multi-functional sensor fusion approach where the radar sensor serves both for detection and velocity measurement, while the camera provides complementary optical flow information. This multi-functionality allows the system to overcome the limitations of individual sensors and achieve robust velocity estimation across diverse scenarios.
2Device complexity
If camera optical flow alone is used for velocity estimation, then device complexity is reduced, but reliability deteriorates under dynamic conditions
Solution Approach 1:
The patent introduces an attention mechanism as an intermediary that selectively integrates radar radial velocity information with camera optical flow measurements. This attention mechanism acts as a mediator that corrects optical flow errors using reliable radar velocity data, particularly under dynamic conditions where optical flow alone fails.
Solution Approach 2:
The system implements feedback through the attention mechanism that continuously adjusts the weighting between optical flow and radar velocity based on their respective reliability. When optical flow becomes unreliable under dynamic conditions, the attention mechanism increases reliance on radar measurements, creating a feedback loop that maintains estimation reliability.
3Measurement precision
If multi-sensor fusion is implemented, then measurement precision is improved, but device complexity increases due to multiple sensors and processing requirements
Solution Approach 1:
The patent replaces complex mechanical sensor fusion systems with a computational attention mechanism that fuses radar and camera data. Instead of using multiple complex sensors with dedicated processing hardware for each, the system uses a neural network-based attention mechanism that efficiently integrates multi-sensor data through learned features, reducing overall system complexity while maintaining high precision.
4Reliability
If radar and camera fusion is implemented, then reliability is improved, but loss of information increases due to data association challenges
Solution Approach 1:
The patent addresses data association challenges by transforming the association problem into a different dimensional space using attention mechanisms. Instead of directly associating radar points with camera pixels in 2D space, the attention mechanism operates in feature space, comparing learned features from both sensors to establish correspondences, thereby reducing information loss during association.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
Enhances accuracy and robustness of object velocity estimation by leveraging the complementary strengths of camera and radar sensors, addressing individual limitations and improving trajectory predictions.
Implementation Method 1
ranging sensor information including respective radial velocities of one or more objects relative to a ranging sensor
Data Source
AI summary
An apparatus is configured to generate first image feature vectors for a first frame of the video data, generate second image feature vectors for a second frame of the video data, determine, from first image feature vectors and the second image feature vectors, respective initial 3D velocities of points of the first frame of the video data, generate ranging feature vectors from the ranging sensor information, the ranging sensor information including respective radial velocities of one or more objects relative to a ranging sensor, and respective ranges of the one or more objects relative to the ranging sensor, associate feature vectors from the first image feature vectors and the ranging feature vectors that are from common objects of the one or more objects to generate associated feature vectors, and determine respective output 3D object velocities for the one or more objects based on the associated feature vectors.