Asynchronous LiDAR Camera Fusion via Temporal Hidden States
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Synchronous data fusion of LiDAR and camera data in autonomous driving systems results in high latency due to camera processing delays, while asynchronous fusion with significant timestamp mismatches leads to poor fusion results.
Innovation Solution
A method and apparatus for asynchronous data fusion using temporal fusion nets and depth completion processes to generate fused hidden states from current LiDAR and previous camera data, reducing latency and improving fusion effectiveness by performing temporal fusion based on previous frame data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If synchronous data fusion of LiDAR and camera is performed, then data consistency is improved, but processing latency increases due to camera processing delays
Solution Approach 1:
The patent applies preliminary action by performing LiDAR data processing and generating three-dimensional embeddings before camera data is ready. The system prepares LiDAR features in advance and stores them, so that when camera data becomes available, the fusion can proceed immediately without waiting for LiDAR processing, thus reducing overall latency while maintaining data consistency.
Solution Approach 2:
The patent implements dynamics by creating a dynamic fusion architecture that adapts to the asynchronous arrival of sensor data. The system uses a temporal fusion network that can dynamically combine LiDAR and camera features at different time stamps, allowing the fusion process to be flexible rather than rigidly synchronous, thereby reducing latency while preserving data consistency through temporal alignment.
2Loss of time
If asynchronous data fusion with significant timestamp mismatches is performed, then processing latency is reduced, but fusion effectiveness deteriorates due to data mismatch
Solution Approach 1:
The patent uses an intermediary approach by introducing a temporal fusion network that acts as a mediator between asynchronous LiDAR and camera data. This network processes and aligns features from different time stamps, bridging the timestamp gap through learned temporal relationships, thus maintaining fusion effectiveness despite asynchronous input while enabling lower latency processing.
Solution Approach 2:
The patent applies parameter changes by transforming the fusion problem from requiring synchronized time stamps to working with asynchronous time stamps. The system changes the temporal parameter handling by using recurrent neural networks and temporal attention mechanisms that can process variable time intervals, converting the constraint of synchronous fusion into a flexible asynchronous framework that maintains effectiveness.
3Reliability
If camera processing is performed completely before fusion, then data quality is improved, but overall system latency increases
Solution Approach 1:
The patent applies preliminary action by performing LiDAR data processing and generating three-dimensional embeddings before camera data is ready. The system prepares LiDAR features in advance and stores them, so that when camera data becomes available, the fusion can proceed immediately without waiting for LiDAR processing, thus reducing overall latency while maintaining data consistency.
Solution Approach 2:
The patent implements continuity of useful action by maintaining a continuous processing pipeline where LiDAR data is processed continuously and stored in a buffer. This allows the system to continuously prepare features without interruption, and when camera data arrives, the fusion can immediately utilize the pre-prepared LiDAR features, ensuring continuous productive operation rather than idle waiting periods.
Data Source
AI summary
A method and an apparatus for asynchronous data fusion, a storage medium and an electronic device are provided. The method includes: obtaining current frame LiDAR data, and determining current frame LiDAR three-dimensional embeddings; determining a previous frame fused hidden state, and performing a temporal fusion process based on the previous frame fused hidden state and the current frame LiDAR three-dimensional embeddings to generate a current frame temporary hidden state and a current frame output result; and obtaining current frame camera data, determining current frame camera three-dimensional embeddings, and generating a current frame fused hidden state based on the current frame camera three-dimensional embeddings and the current frame temporary hidden state. Asynchronous fusion is performed on the current frame LiDAR data and previous frame camera data, which leads to a low processing latency.