Event-Based Sensor Data Structure for Connected Component Labeling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computer vision algorithms, such as connected component labeling (CCL), are not suitable for processing event-based data from event-based sensors, which limits their ability to effectively extract and track features in high dynamic range and fast-moving scenarios, leading to computational inefficiencies and unnecessary overhead.
Innovation Solution
A method for processing event-based data involves storing a data structure for each pixel with recent event information and applying connectivity criteria to label and cluster events, allowing for online grouping and feature extraction without the need for redundant data processing, using criteria like timestamp, polarity, flickering frequency, visual flow, and semantic IDs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If framed-based image sensors are used to capture visual information, then the system can process images with standard computer vision algorithms, but the system fails to render fast motion and cannot track objects with high dynamic resolution
Solution Approach 1:
The patent changes the fundamental parameter of image capture from temporal sampling (fixed frame rate) to asynchronous event-based capture. Each pixel independently generates events based on luminance changes, enabling unlimited temporal resolution without frame rate constraints. This resolves the contradiction by allowing high dynamic resolution tracking of fast motion while maintaining computational efficiency through event-driven processing.
Solution Approach 2:
The system transitions from static frame-based processing to dynamic event-based processing. Instead of processing complete frames at fixed intervals, the system processes only the events (luminance changes) that occur in real-time. This dynamic approach enables the system to adapt its processing rate to the actual scene changes, achieving high measurement precision for fast motion without the productivity penalty of processing every frame at high frame rates.
2Measurement precision
If framed-based image sensors operate at very high frame rates to track fast moving objects, then tracking precision improves, but computational intensity increases significantly
Solution Approach 1:
The patent extracts only the essential information (luminance changes as events) from the continuous visual stream, rather than processing every frame. By taking out only the meaningful changes and ignoring redundant unchanged regions, the system achieves high tracking precision for fast moving objects without the computational intensity of processing complete high-frame-rate frames. This extraction approach reduces computational load while maintaining precision.
Solution Approach 2:
Instead of processing the entire frame at high frame rates (excessive action), the system processes only the partial events that represent actual changes (luminance transitions). This partial action approach processes only the necessary information for tracking, achieving high precision without the excessive computational intensity of full frame processing at high frame rates.
3Loss of information
If framed-based image sensors process redundant frames in the absence of scene change, then complete image information is maintained, but computational power is wasted
Solution Approach 1:
The system maintains continuous processing readiness through event-driven architecture. Instead of periodically processing complete frames (which wastes energy when nothing changes), the system remains in a low-power state until an event occurs. When events occur, processing activates immediately. This continuity of useful action (processing only when needed) eliminates the energy waste of processing redundant frames while maintaining information completeness through real-time event processing.
Solution Approach 2:
The system replaces periodic frame-based processing with event-triggered processing. Instead of processing at fixed periodic intervals (which includes redundant frames), the system processes only when events occur. This transforms the action from periodic to event-driven, eliminating computational waste on redundant frames while maintaining complete information through real-time event capture and processing.
4Ease of manufacture
If connected component labeling algorithms are applied to event-based data, then feature extraction can be performed, but the algorithms are not suitable for the asynchronous event stream format
Solution Approach 1:
The patent adapts connected component labeling to handle dynamic event streams asynchronously. Instead of processing static images, the algorithm processes events in real-time as they arrive, maintaining a dynamic data structure that tracks connected components across time. This dynamic adaptation preserves the productivity benefits of CCL while making it suitable for asynchronous event-based data formats.
Solution Approach 2:
The system changes the input parameter format from complete frames to individual events. By modifying how CCL processes input (from frame-based to event-based), the algorithm becomes applicable to event streams while maintaining its core functionality. This parameter change enables efficient processing of asynchronous events without sacrificing the productivity advantages of connected component labeling.
Data Source
AI summary
An event-based sensor has an array of pixels facing a scene. The information to be processed includes events originating from the pixels depending on variations of incident light from the scene. A method includes storing a data structure for a set of pixels of the array, the data structure including, for each pixel of the set, event data associated with a most recent event originating from the pixel. Upon receiving a current event from a first pixel of the array, the method includes retrieving any event data included in the data structure for a group of pixels including the first pixel and a plurality of second pixels adjacent to the first pixel in the array. The current event is labelled based on at least one connectivity criterion between the current event and the most recent event originating from a pixel of the group.