Temporal Feature Map Fusion for Autonomous Perception
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional machine learning models for autonomous vehicles face challenges in processing temporal sensor data efficiently, leading to high computing resource requirements and latency, making them less suitable for real-time applications.
Innovation Solution
The system preprocesses feature maps from previous timestamps using temporal transformation layers before fusion with current feature maps, allowing for parallel processing to reduce latency and improve temporal understanding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If temporal fusion of feature maps from multiple timestamps is performed using conventional methods (aggregation by aligning and concatenating features or query-based sampling), then temporal understanding of the environment is improved, but computing resource requirements increase and latency is introduced
Solution Approach 1:
The system performs preprocessing of feature maps from previous timestamps in advance, transforming them into a reference coordinate frame before fusion with current feature maps. This preliminary transformation enables more efficient fusion operations and reduces latency during real-time processing.
Solution Approach 2:
The temporal fusion process is divided into distinct stages: preprocessing of historical feature maps, coordinate frame transformation, and fusion with current frames. This segmentation allows parallel processing of different timestamped feature maps, improving computational efficiency and reducing overall latency.
2Measurement precision
If a large number of frames are processed and fused temporally, then temporal understanding accuracy is improved, but processing time and latency increase
Solution Approach 1:
Feature maps from previous timestamps are preprocessed and transformed into the reference coordinate frame in advance, before they are needed for fusion. This preliminary action reduces the computational burden during real-time fusion operations, enabling faster processing of multiple frames while maintaining temporal accuracy.
Solution Approach 2:
The system processes and transforms feature maps from multiple timestamps (excessive action) but implements efficient fusion strategies that selectively combine the most relevant temporal information. This allows the system to maintain high temporal understanding accuracy while avoiding the full computational cost of processing every possible frame combination.
3Loss of time
If feature maps are preprocessed using temporal transformation layers in parallel with current feature map processing, then latency is reduced, but device complexity increases
Solution Approach 1:
The machine learning model is divided into separate functional layers: temporal transformation layers for preprocessing historical feature maps, and fusion layers for combining transformed features with current frames. This segmentation allows parallel execution of transformation and fusion operations, reducing latency while maintaining manageable model complexity through modular design.
Solution Approach 2:
The temporal transformation layers serve multiple functions: they transform coordinate frames, align temporal features, and prepare historical feature maps for fusion. This multi-functionality reduces the need for separate specialized components, thereby reducing overall device complexity while enabling parallel processing to reduce latency.
Data Source
AI summary
In various examples, temporal-based perception for autonomous or semi-autonomous systems and applications is described. Systems and methods are disclosed that use a machine learning model (MLM) to intrinsically fuse feature maps associated with different sensors and different instances in time. To generate a feature map, image data generated using image sensors (e.g., cameras) located around a vehicle are processed using a MLM that is trained to generate the feature map. The MLM may then fuse the feature maps in order to generate a final feature map associated with a current instance in time. The feature maps associated with the previous instances in time may be preprocessed using one or more layers of the MLM, where the one or more layers are associated with performing temporal transformation before the fusion is performed. The MLM may then use the final feature map to generate one or more outputs.