Recurrent Neural Network Object Detection Using Concatenated Feature Maps
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current object detection systems in autonomous vehicles struggle with leveraging temporal information and handling poor lighting conditions, leading to unstable object localization and failure in detecting objects when they are occluded or in challenging environmental conditions.
Innovation Solution
The use of recurrent connections within neural networks to generate and utilize concatenated feature maps from multiple sensor types, such as RGB and depth maps, to incorporate temporal information and improve object detection by maintaining state information and enhancing feature extraction from previous frames.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If object detection systems use conventional single-frame processing, then the system complexity is low, but object detection accuracy and reliability deteriorate in poor lighting and occlusion conditions
Solution Approach 1:
The system performs preliminary feature extraction from multiple sensor types (RGB, depth, infrared) and stores them as feature maps before detection is needed. When detecting objects in challenging conditions, these pre-extracted features from previous frames are reused and combined, avoiding the need to reprocess raw sensor data and improving reliability without proportionally increasing real-time complexity
Solution Approach 2:
Feature maps serve as intermediary representations between raw sensor data and final object detection results. These intermediate feature maps capture essential object characteristics in multiple modalities and can be combined across time and sensor types, enabling reliable detection in poor lighting by mediating between different sensor inputs and the detection algorithm
2Loss of information
If the system processes only current frame data, then processing speed is high, but temporal information and object tracking capability are lost
Solution Approach 1:
The system merges feature maps from multiple sensor types (RGB, depth, infrared) and multiple time frames into a combined feature representation. This merging occurs at the feature level rather than raw data level, efficiently integrating temporal information and multi-sensor data while avoiding redundant processing of complete frames, thus retaining temporal information without proportionally increasing processing time
Solution Approach 2:
The processing is segmented into separate feature extraction stages for different sensor types, with each sensor stream processed independently to generate its own feature map. These segmented feature maps are then combined, allowing selective integration of temporal information from each sensor modality while maintaining processing efficiency
3Measurement precision
If single sensor type is used for detection, then device complexity is low, but detection accuracy in poor lighting conditions deteriorates
Solution Approach 1:
The system implements a universal detection framework that processes multiple sensor types (RGB, depth, infrared) through a common feature extraction and object detection pipeline. Each sensor type contributes complementary information that improves detection accuracy in various conditions, with the system automatically selecting and combining relevant features from each sensor modality
Solution Approach 2:
The system creates a composite feature representation by combining features from multiple sensor types in the same way composite materials combine different substances to achieve superior properties. The concatenated feature maps integrate RGB color information, depth spatial information, and infrared thermal information to form a robust multi-modal feature set that improves detection accuracy in poor lighting conditions
Data Source
AI summary
According to one embodiment, a system includes a sensor component and a detection component. The sensor component is configured to obtain a first stream of sensor data and a second stream of sensor data, wherein each of the first stream and second stream comprise a plurality of sensor frames. The detection component is configured to generate a concatenated feature map based on a sensor frame of a first type and a sensor frame of a second type. The detection component is configured to detect one or more objects based on the concatenated feature map. One or more of generating and detecting comprises generating or detecting using a neural network with a recurrent connection that feeds information about features or objects from previous frames.