Virtual Sensor Mapping for Scalable 3D Perception Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for training neural networks for autonomous driving are not scalable due to the need for separate models for each sensor setup, which is time-consuming and expensive, and are sensitive to sensor-specific properties, leading to inaccuracies in depth estimation.
Innovation Solution
A method involving sensor virtualization, where physical sensor data from multiple modalities is mapped to virtual sensors with predefined characteristics, allowing for a unified training process in a 3D model space, enabling the combination of data from different sensors and improving reliability by compensating for sensor failures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If separate neural network models are trained for each sensor setup, then the model can be optimized for specific sensor characteristics, but the training process becomes time-consuming and expensive with poor scalability
Solution Approach 1:
The patent introduces a unified neural network model that can process multiple sensor modalities (camera, LiDAR, radar) through a common architecture. The model uses modality-specific processing branches that feed into shared feature extraction and prediction layers, allowing a single model to handle different sensor setups without requiring separate training for each configuration.
Solution Approach 2:
The patent employs learnable projection matrices and transformation parameters that adapt the input features from different sensor modalities to a unified representation space. These parameters are learned during training, enabling the model to adjust to various sensor characteristics without retraining the entire network for each sensor type.
2Measurement precision
If separate neural network models are trained for each sensor setup, then the model can be optimized for specific sensor characteristics, but the training cost increases significantly
Solution Approach 1:
The unified model architecture processes multiple sensor modalities through shared computational layers, reducing the total number of training operations required. Instead of training separate models for each sensor type, the system trains one model that leverages data from all modalities, significantly reducing computational costs and energy consumption.
Solution Approach 2:
The patent merges the processing pipelines for different sensor modalities into a single unified network. The modality-specific features are combined in intermediate layers, allowing the system to learn from all sensor types simultaneously rather than requiring separate training processes for each modality.
3Reliability
If physical sensor data is used directly without virtualization, then the data reflects actual sensor characteristics, but the system becomes sensitive to sensor-specific properties and lacks adaptability to different sensor setups
Solution Approach 1:
The patent introduces a virtual sensor representation as an intermediary layer between physical sensors and the neural network. This virtual representation standardizes the input data format and characteristics, acting as a mediator that preserves the essential information from physical sensors while presenting a unified, sensor-agnostic interface to the processing network.
Solution Approach 2:
The patent segments the processing into modality-specific feature extraction followed by unified processing. Each sensor modality is processed through its own feature extraction branch that captures modality-specific characteristics, then these features are segmented and combined in a unified representation that is independent of the original sensor setup.
4Productivity
If a unified neural network model is used for multiple sensor modalities, then training efficiency improves and scalability increases, but the model must handle heterogeneous data formats and characteristics
Solution Approach 1:
The unified model is segmented into modality-specific processing branches and shared processing layers. Each branch handles the unique characteristics of its sensor type (camera, LiDAR, radar) independently, then feeds into common feature fusion and prediction layers. This segmentation manages complexity by localizing modality-specific processing while maintaining unified architecture benefits.
Solution Approach 2:
The patent applies local quality by making different parts of the network specialized for different modalities. The input layers and early feature extraction layers are modality-specific, adapting to local data characteristics, while deeper layers use shared weights and operations that benefit from multi-modal data. This local specialization reduces the complexity burden on any single part of the system.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
The present invention provides a method for training a neural network to predict objects in a surrounding of a vehicle, the method comprising: - obtaining physical sensor data from physical sensors having one or more modalities, - performing a first mapping from the physical sensor data to virtual sensors to obtain virtual sensor data, - performing a second mapping from the virtual sensor data to a 3D model space, and - training the neural network based on the virtual sensor data and one or more annotations in the 3D model space.