3D Object Detection With Segmented Temporal Frame Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current 3D object detection systems in autonomous vehicles face challenges in improving time, quality, and computing resource efficiency when using temporal histories of image frames.
Innovation Solution
A system that fuses multiple image frames into a feature map using neural networks to generate 3D objects, incorporating both short-term and long-term fusion models to enhance detection accuracy and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If temporal histories of image frames are used for 3D object detection, then detection accuracy is improved, but computing resource consumption increases
Solution Approach 1:
The patent segments the temporal history processing into two distinct streams: a short-term fusion model that processes recent frames (t, t-1, t-2) and a long-term fusion model that processes historical frames (t-3 to t-k). This segmentation allows the system to handle different temporal scales separately, improving detection accuracy for both recent and historical objects while managing computational resources by processing only the necessary number of frames for each temporal category.
Solution Approach 2:
The patent applies partial action by selectively processing only the necessary number of temporal frames for each fusion model. The short-term model uses a limited number of recent frames (3 frames), while the long-term model uses a limited number of historical frames (k frames where k < infinity). This partial processing approach maintains detection accuracy for the required temporal scope while significantly reducing computational resource consumption compared to processing all historical frames.
2Measurement precision
If multiple image frames are fused to improve detection, then object detection quality improves, but processing time increases
Solution Approach 1:
The patent segments the temporal fusion process into two independent parallel streams: short-term fusion and long-term fusion. Each stream processes its specific temporal window independently, allowing simultaneous computation. The short-term fusion model processes frames t, t-1, t-2 while the long-term fusion model processes frames t-3 to t-k in parallel, reducing total processing time compared to sequential processing of all frames.
Solution Approach 2:
The patent applies partial action by limiting the number of frames processed in each temporal stream. Instead of processing all historical frames sequentially, the long-term model processes only k historical frames (where k is a finite number less than the total number of available frames). This partial processing maintains sufficient detection quality for historical object detection while significantly reducing processing time.
3Measurement precision
If temporal data is processed for 3D object detection, then detection accuracy improves, but device complexity increases
Solution Approach 1:
The patent segments the temporal data processing system into two distinct modules: a short-term fusion model and a long-term fusion model. Each module has a specific temporal scope and processing function. This segmentation simplifies the overall system architecture by dividing the complex task of processing all historical frames into two manageable, independent streams, each with well-defined input and output characteristics.
Solution Approach 2:
The patent applies universality by designing both fusion models to process the same types of input data (image frames) and produce the same type of output (3D object detection results), but with different temporal scopes. The short-term model handles recent frames while the long-term model handles historical frames, but both use the same fundamental processing architecture, reducing overall system complexity through code reusability and architectural consistency.
Data Source
AI summary
Apparatuses, systems, and techniques of using one or more machine learning processes (e.g., neural network(s)) to detect objects from a plurality of image frames. In at least one embodiment, a plurality of image frames are fused into a feature map using one or more neural networks. In at least one embodiment, a plurality of image frames are processed using one or more neural networks to detect objects in a 3D space.