Vehicle Sensor Fusion with Interpolated Frames for Unsynced Sensors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The fusion of unsynchronized data frames from multiple sensors in vehicles, such as cameras, radars, and lidars, leads to errors in creating a three-dimensional model of the environment due to differences in frame capture times, making it challenging to synchronize data from sensors based on different technologies.
Innovation Solution
A sensor data fusion system that synthesizes an interpolated-frame with a specific timestamp to align data from different sensors, using motion-flow-analysis or other image processing techniques to create a temporally synchronized image or point-cloud, allowing for effective fusion and accurate 3D model rendering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple sensors with different frame rates are used to capture environmental data, then the coverage and detection capability are improved, but temporal synchronization errors are introduced
Solution Approach 1:
The system performs preliminary actions by capturing and storing multiple consecutive frames from the first sensor before the fusion operation. These pre-captured frames (including previous, current, and next frames) are then used to synthesize an interpolated frame that matches the timestamp of the second sensor, eliminating the need for real-time synchronization during fusion.
Solution Approach 2:
The patent introduces an interpolated frame as an intermediary element between the two unsynchronized sensors. This intermediate frame, synthesized from the first sensor's multiple frames, serves as a bridge that matches the temporal characteristics of the second sensor, enabling accurate fusion without requiring direct synchronization between the original sensors.
2Measurement precision
If frame interpolation is performed to synchronize timestamps, then temporal alignment accuracy is improved, but computational complexity increases
Solution Approach 1:
The system applies partial action by selectively using only the necessary frames for interpolation (previous, current, and next frames) rather than processing all available data. The interpolation focuses specifically on generating the single required frame that matches the second sensor's timestamp, avoiding unnecessary computational overhead while achieving precise temporal alignment.
3Measurement precision
If multiple frames are stored for interpolation, then interpolation accuracy is improved, but memory requirements increase
Solution Approach 1:
The system applies local quality by storing and processing only the specific frames locally required for interpolation (the immediate previous, current, and next frames) rather than maintaining all historical data. This localized approach ensures sufficient interpolation accuracy while minimizing memory consumption to only what is strictly necessary for the current fusion operation.
Data Source
AI summary
A sensor data fusion system for a vehicle with multiple sensors includes a first-sensor, a second-sensor, and a controller-circuit. The first-sensor is configured to output a first-frame of data and a subsequent-frame of data indicative of objects present in a first-field-of-view. The first-frame is characterized by a first-time-stamp, the subsequent-frame of data characterized by a subsequent-time-stamp different from the first-time-stamp. The second-sensor is configured to output a second-frame of data indicative of objects present in a second-field-of-view that overlaps the first-field-of-view. The second-frame is characterized by a second-time-stamp temporally located between the first-time-stamp and the subsequent-time-stamp. The controller-circuit is configured to synthesize an interpolated-frame from the first-frame and the subsequent-frame. The interpolated-frame is characterized by an interpolated-time-stamp that corresponds to the second-time-stamp. The controller-circuit fuses the interpolated-frame with the second-frame to provide a fused-frame of data characterized by the interpolated-time-stamp, and operates the host-vehicle in accordance with the fused-frame.