Event Sensor Image Segmentation for Autonomous Navigation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional semantic image segmentation methods using only RGB sensors struggle with accurately distinguishing objects in scenes with shadows, glare, and transparent regions, leading to false classifications and erroneous navigation in autonomous vehicles and other applications.
Innovation Solution
A method and apparatus for real-time semantic image segmentation using a combination of RGB and monocular event-based sensors, where event velocity is calculated by correlating event data across frames, and semantic labeling is performed when the event velocity exceeds a threshold, enabling accurate identification of objects and motion information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional semantic image segmentation methods using only RGB sensors are employed, then the system structure remains simple, but the accuracy of object classification deteriorates in scenes with shadows, glare, and transparent regions
Solution Approach 1:
The patent combines RGB sensors and event-based sensors into a unified sensor system. The RGB sensor captures color information while the event-based sensor detects changes in luminance over time. By merging the data from both sensor types, the system achieves superior object classification accuracy in challenging conditions (shadows, glare, transparent regions) compared to using either sensor alone, while maintaining reasonable system complexity through efficient data fusion algorithms.
2Reliability
If semantic labeling is performed on every RGB frame, then complete scene understanding is achieved, but the processing time and computational load increase significantly
Solution Approach 1:
Instead of performing semantic labeling on every RGB frame continuously, the system uses event-based triggers to initiate labeling only when significant changes occur in the scene. The event-based sensor detects luminance changes and generates events only when thresholds are exceeded, transforming the continuous periodic labeling task into an event-driven intermittent labeling process. This maintains complete scene understanding for dynamic regions while dramatically reducing processing time and computational load for static scenes.
3Measurement precision
If event velocity threshold filtering is applied, then false positives are reduced, but some dynamic regions may be missed
Solution Approach 1:
The system applies different processing strategies to different regions of the scene based on their characteristics. For regions with high event velocity exceeding the threshold, full semantic labeling is performed to ensure accurate detection of significant dynamic objects. For regions with low event velocity below the threshold, the system uses predictive models to maintain object tracking without intensive processing. This local differentiation ensures that false positives are reduced in static regions while motion information completeness is preserved in dynamic regions through adaptive processing.
Data Source
Figure 1A
Figure 1B
Figure 1C
AI summary
A method for real-time semantic image segmentation using a monocular event-based sensor includes capturing a scene using a red, green, blue (RGB) sensor to obtain a plurality of RGB frames and an event sensor to obtain event data corresponding to each of the plurality of RGB frames, performing object labeling for objects in a first RGB frame among the plurality of RGB frames by identifying one or more object classes, obtaining an event velocity of the scene by fusing the event data corresponding to the first RGB frame and at least one subsequent RGB frame among the plurality of RGB frames, determining whether the event velocity is greater than a predefined threshold, and performing object labeling for objects in the at least one subsequent RGB frame based on the determination.