Student Model Vector Map Learning via Cross-Correlation Loss
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing camera-based map generation technologies struggle with the accuracy of representing three-dimensional physical elements due to the use of primarily two-dimensional information, and the integration of lidar and camera sensors increases costs and reduces real-time inference capabilities.
Innovation Solution
A method and apparatus for generating vector maps using a student model that learns by back-propagating cross-correlation loss, where the student model infers features from camera data and replicates cross-correlations with features extracted from multi-modal data including lidar, to improve the accuracy and efficiency of map generation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If camera-based systems are used for map generation, then costs are reduced compared to lidar-based systems, but the accuracy of representing three-dimensional physical elements deteriorates
Solution Approach 1:
The patent uses a student model that copies knowledge from a teacher model trained on multi-modal data. The student model learns to generate vector maps from camera data alone by replicating the cross-correlation relationships between camera features and lidar features that were learned during teacher model training, thereby achieving accurate 3D representation without requiring expensive lidar hardware
Solution Approach 2:
The patent introduces an intermediary teacher model that was trained on both camera and lidar data. This teacher model serves as a mediator that transfers the complex cross-correlation relationships between different sensor modalities to the student model, enabling the student model to achieve accurate 3D mapping using only camera data
2Measurement precision
If both camera and lidar sensors are used for map generation, then the accuracy of predicting physical elements is improved, but the system becomes expensive and slow for real-time inference
Solution Approach 1:
The patent extracts only the essential cross-correlation relationships between camera and lidar features from the multi-modal data, and encodes them into the student model's weight parameters. This extraction allows the student model to achieve accurate physical element prediction using only camera data, eliminating the need for real-time processing of multiple sensor modalities and enabling real-time inference
Solution Approach 2:
The patent performs preliminary training of the teacher model on both camera and lidar data during the training phase. The complex cross-correlation relationships are pre-computed and stored in the teacher model's parameters, which are then transferred to the student model. This preliminary action removes the computational burden from real-time inference, allowing the student model to process only camera data quickly
Data Source
AI summary
A learning method of generating a vector map and a method and apparatus for generating a vector map are disclosed. The learning method includes converting a first feature extracted by inputting a first modality sensed by a first sensor to a student model into a first feature vector in a bird eye view (BEV) space, converting a second feature extracted by inputting a multi-modality including the first modality and a second modality sensed by a second sensor to a teacher model into a second feature vector in the BEV space, and learning the student model to generate a vector map corresponding to the first modality by back-propagating cross-correlation loss by dimension that causes the first feature vector to replicate a cross-correlation with the second feature vector to the student model.