Fused Sensor Data via Metadata Association
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional techniques face challenges in fusing 3D point cloud data from LiDAR sensors with 2D image data from cameras in autonomous vehicles due to lack of correspondence between metadata labels, making it difficult to generate fused sensor data for training machine learning models.
Innovation Solution
A method involving frame synchronization and metadata association algorithms is employed to align and fuse 3D point cloud data with 2D image data by calibrating extrinsics and determining similarity scores between labels, creating a richer training dataset for machine learning models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If metadata is associated with sensor data at different points in time, then each sensor can independently process and label its data, but correspondence between sensor data from different sensors cannot be established
Solution Approach 1:
The system performs preliminary frame synchronization by associating timestamps with each sensor frame and sorting frames based on these timestamps before metadata association. This preliminary temporal ordering ensures that when metadata is subsequently associated with sensor data, corresponding frames from different sensors are already aligned, preserving spatial-temporal correspondence while allowing independent processing.
2Ease of operation
If conventional techniques are used to capture and process sensor data, then individual sensor data can be obtained, but fused sensor data with established correspondence between different sensors cannot be generated
Solution Approach 1:
The system segments the sensor data fusion process into distinct modular steps: frame capture with timestamp association, frame sorting based on timestamps, metadata association with synchronized frames, and fused data generation. This segmentation allows each module to operate independently and simplifies the overall complex process of multi-sensor data fusion while maintaining correspondence between sensors.
3Productivity
If no frame synchronization is performed, then processing is simpler and faster, but accurate correspondence between 3D point cloud data and 2D image data cannot be established
Solution Approach 1:
The system changes the temporal parameter of sensor data by associating precise timestamps with each captured frame and then sorting frames based on these timestamp parameters. This parameter-based synchronization approach enables accurate spatial correspondence between 3D point cloud data and 2D image data while maintaining efficient processing through automated timestamp-based ordering rather than complex real-time coordination.
Data Source
AI summary
Described herein are systems, methods, and non-transitory computer readable media for generating fused sensor data through metadata association. First sensor data captured by a first vehicle sensor and second sensor data captured by a second vehicle sensor are associated with first metadata and second metadata, respectively, to obtain labeled first sensor data and labeled second sensor data. A frame synchronization is performed between the first sensor data and the second sensor data to obtain a set of synchronized frames, where each synchronized frame includes a portion of the first sensor data and a corresponding portion of the second sensor data. For each frame in the set of synchronized frames, a metadata association algorithm is executed on the labeled first sensor data and the labeled second sensor data to generate fused sensor data that identifies associations between the first metadata and the second metadata.