RGB + Event Sensor Fusion for Robust AV Object Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current autonomous vehicle (AV) object detection systems struggle with low reliability under varying lighting and environmental conditions, such as nighttime or adverse weather, due to limitations in dynamic range and motion blur in conventional camera systems, and event-based cameras fail in static views.
Innovation Solution
A deep learning method combining inputs from both static-based cameras and event-based cameras using a convolutional neural network architecture to enhance object detection and classification, particularly in challenging conditions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional cameras are used for object detection, then the system can detect objects in static scenes, but the detection reliability drops under varying lighting conditions and the system suffers from low dynamic range and motion blur
Solution Approach 1:
The patent combines event-based camera data with conventional frame-based camera data into a unified neural network architecture. The event camera provides high-temporal-resolution motion information that complements the spatial information from conventional cameras, creating a robust multi-modal system that maintains reliability under varying lighting conditions while avoiding the limitations of either sensor type alone
2Speed
If event-based cameras are used for object detection, then the system achieves high temporal resolution and high dynamic range, but the detection fails in static views where no motion events occur
Solution Approach 1:
The patent merges event-based camera streams with frame-based camera streams in a joint detection framework. The conventional camera provides reliable static scene information while the event camera contributes high-temporal-resolution motion data, creating a complementary system where each sensor type compensates for the other's weaknesses
Solution Approach 2:
The patent implements a dynamic fusion approach where the neural network adaptively processes event data and frame data based on scene conditions. The system dynamically adjusts to utilize event-based high-temporal-resolution data when motion is present and falls back on frame-based data for static scenes, maintaining reliable detection across all conditions
3Adaptability or versatility
If separate event-based and frame-based streams are processed independently, then each stream can be optimized individually, but the system complexity increases and the integration of complementary information is lost
Solution Approach 1:
The patent merges event-based and frame-based streams into a unified convolutional neural network architecture that processes both data types together from the early stages. This joint processing approach maintains the complementary information from both sensor types while using a shared architecture to reduce overall system complexity compared to fully independent processing pipelines
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Techniques are disclosed to facilitate, in autonomous vehicles, the robust detection and classification of objects in a scene using a static sensors in conjunction with event-based sensors. A trained system architecture may be implemented, and the fusion of both sensors thus allows for the consideration of scenes with overexposure, scenes with underexposure, as well as scenes in which there is no movement. In doing so, the autonomous vehicle may detect and classify objects in conditions in which each sensor, if operating separately, would not otherwise be able to classify (or classify with high uncertainty) due to the sensing environment.