Fused Sensor Graphs for Lossless Automotive Perception
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for processing sensor data from multiple automotive sensors in vehicles lead to information loss and increased computational effort due to the need for grid formation and limited inclusion of relative distances and locational information from all sensors.
Innovation Solution
A method for generating a fused automotive sensor data graph by combining sensor data from multiple sensors using calibration matrices to create a connected graph that retains all sensor data information, reducing the need for multiple training sessions and computational resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If sensor data from multiple automotive sensors is combined in a two-dimensional grid or three-dimensional grid, then joint processing of sensor data using image processing approaches is enabled, but storage requirements increase and processing times are prolonged due to empty cells that need to be processed
Solution Approach 1:
The patent segments the sensor data processing by creating individual sensor data graphs for each sensor type (camera, radar, LIDAR) separately, then combining them through node enrichment rather than creating a single large grid. This segmentation avoids processing empty cells while maintaining joint processing capability through the graph combination approach.
Solution Approach 2:
The patent introduces an intermediary representation - the sensor data graph - that mediates between individual sensor data and the final fused perception. The graph serves as a compact intermediate structure that preserves relative distance information without requiring a complete grid representation, thus reducing processing time while enabling joint processing.
2Ease of operation
If sensor data from multiple automotive sensors is combined in a two-dimensional grid or three-dimensional grid, then joint processing of sensor data using image processing approaches is enabled, but storage requirements increase due to empty cells
Solution Approach 1:
The patent segments the sensor data processing by creating individual sensor data graphs for each sensor type (camera, radar, LIDAR) separately, then combining them through node enrichment rather than creating a single large grid. This segmentation avoids storing empty cells while maintaining joint processing capability through the graph combination approach.
Solution Approach 2:
The patent extracts only the relevant sensor data points that contain actual information, organizing them into graph nodes based on detected objects. This extraction approach removes empty cells from the representation, significantly reducing storage requirements while preserving the essential sensor data for joint processing.
3Ease of operation
If sensor data from multiple automotive sensors is combined in a two-dimensional grid or three-dimensional grid, then joint processing of sensor data using image processing approaches is enabled, but information loss occurs due to discretization of sensor data
Solution Approach 1:
The patent uses a dynamic graph structure where nodes and edges can be added or removed based on detected objects and their relationships. This dynamic approach allows the representation to adapt to the actual scene content without fixed grid discretization, preserving continuous spatial information and reducing information loss while still enabling structured joint processing.
4Adaptability or versatility
If each node of the concatenated automotive sensor graph includes sensor data of various sensors and thus various sensor types, then comprehensive data integration is achieved, but training the machine model requires multiple training sessions
Solution Approach 1:
The patent applies local quality by creating separate sensor data graphs for different sensor types with their specific characteristics, then combining them through node enrichment. This allows the machine learning model to process each sensor type's data with appropriate local processing rules, reducing training complexity compared to handling all sensor types uniformly in a single graph.
5Loss of information
If the concatenated automotive sensor graph is formed by adding sensor data of other sensors to nodes of the initial sensor data graph, then information enrichment is achieved, but computational effort increases
Solution Approach 1:
The patent segments the computational process by maintaining separate sensor data graphs for different sensor types until the enrichment stage. This segmentation allows for more efficient processing of each sensor type's data independently, reducing overall computational effort compared to processing all sensors simultaneously in a single concatenated graph.
Data Source
AI summary
A method for generating a fused automotive sensor data graph as well as an automotive control unit configured to perform the method are disclosed. A plurality of automotive sensor data graphs is obtained, which are each based on sensor data captured by a corresponding automotive sensor of the vehicle. At least one calibration matrix is obtained, which defines a transformation between sensor data nodes of at least two automotive sensor data graphs. Based on the plurality of automotive sensor data graphs and the at least one calibration matrix, the fused automotive sensor data graph is generated, which includes the plurality of automotive sensor data graphs as well as a plurality of fusion edges. The fused automotive sensor data graph is provided to an automotive perception function, which is implemented by a graph neural network.