Sensor Data Feature Fusion via Spatial Transform Network Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing autonomous vehicle systems face challenges in fusing sensor data from multiple sensors to create a cohesive and spatially consistent bird's eye view (BEV) mapping, which is essential for accurate navigation and path planning.
Innovation Solution
A feature fusion system that uses circuitry to identify and extract features from sensor data from multiple sensors, convert these features into BEV projections, and align them using a spatial transform network (STN) to generate a fused feature map.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If multiple sensors are used to acquire data about the vehicle's external environment, then the quantity and quality of environmental information is improved, but the complexity of fusing this data into a cohesive and spatially consistent dataset increases
Solution Approach 1:
The patent segments the data fusion process into distinct modules: feature extraction modules for each sensor type, BEV projection modules, and alignment modules. Each sensor's data is processed independently through dedicated extraction modules before being integrated, which manages complexity while handling multiple data sources
Solution Approach 2:
The patent introduces BEV (bird's eye view) projection as an intermediary representation that mediates between different sensor modalities. Features from cameras and LIDAR are both projected into the common BEV framework, enabling consistent alignment and fusion without direct complex interactions between heterogeneous sensor data
2Measurement precision
If sensor data from multiple sensors is fused to create a cohesive dataset, then the spatial consistency and accuracy of BEV mapping is improved, but the computational resources and processing time required increase
Solution Approach 1:
The patent performs preliminary feature extraction and BEV projection for each sensor independently before the alignment and fusion steps. This preliminary processing organizes the data in advance, reducing the computational burden during the critical alignment phase and improving overall efficiency
Solution Approach 2:
The patent transforms features from different sensor coordinate systems into a common BEV reference frame, changing the parameter space in which alignment occurs. This coordinate transformation simplifies the fusion process by establishing a unified reference system that reduces computational complexity
3Loss of information
If features are extracted and projected into BEV from multiple sensors, then the completeness of environmental representation is improved, but the difficulty of aligning and fusing these features accurately increases
Solution Approach 1:
The patent employs feedback mechanisms where the alignment model continuously refines the transformation parameters between sensors based on the quality of feature matching in BEV space. This iterative feedback improves alignment accuracy while maintaining complete environmental representation from multiple sensor views
Data Source
AI summary
In one aspect, a method of fusing sensor data from a plurality of sensors is provided. The method comprises identifying first and second sensor data from first and second sensors of the plurality of sensors, respectively, extracting first and second features from the first and second sensor data, respectively, and converting the first and second features to first and second BEV projections of the first and second features, respectively. The method further comprises implementing a spatial transform network that aligns the second BEV projection with the first BEV projection, and generating a fused feature map comprising the aligned second BEV projection and the first BEV projection.