Frame and Event Camera Fusion for Accurate Object Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional frame-based cameras often misdetect or misclassify objects, especially those in motion, leading to inaccuracies in critical applications that rely on object detection and classification.
Innovation Solution
A system and method that combines frame and event camera processing to robustly detect and classify objects by determining object classification tags based on feature points in static frames and pixel-level movement information in event frames, enhancing detection and classification accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional frame-based camera processing is used, then the system is simple to operate, but object detection and classification accuracy deteriorates
Solution Approach 1:
The patent combines frame-based camera processing with event-based camera processing into a unified system. The frame camera captures static images while the event camera detects pixel-level movements asynchronously. By merging the outputs of both cameras and processing them together through a combined neural network, the system achieves superior object detection accuracy for both stationary and moving objects, resolving the contradiction between maintaining simplicity and improving measurement precision.
2Reliability
If conventional frame-based camera processing is used, then the device complexity is low, but reliability in critical applications deteriorates
Solution Approach 1:
The system merges frame-based and event-based processing pipelines, where the frame camera provides comprehensive scene information and the event camera provides motion-sensitive information. This combination enhances detection reliability for critical applications by capturing both static and dynamic objects effectively, while the integrated processing architecture manages complexity through unified neural network processing.
Solution Approach 2:
The system dynamically adapts processing based on scene content by using event camera data to identify regions of motion and directing enhanced processing resources to those areas. This dynamic allocation improves reliability for moving objects while maintaining efficient processing for static scenes, managing complexity through adaptive resource distribution.
3Measurement precision
If frame-based processing alone is used, then energy consumption is low, but detection of moving objects deteriorates
Solution Approach 1:
The event camera operates asynchronously and only generates data when pixel intensity changes exceed a threshold, creating a dynamic data flow that adapts to scene activity. This dynamic operation mode improves motion detection accuracy while consuming energy only when motion occurs, rather than continuously processing all frames regardless of content.
Solution Approach 2:
The event camera provides continuous monitoring of pixel intensities and generates events asynchronously as changes occur, maintaining continuous useful action for motion detection without the energy overhead of continuous frame capture and processing. This continuous event stream ensures moving objects are detected reliably while optimizing energy consumption based on actual scene dynamics.
Data Source
AI summary
A system and method for object classification and related applications based on frame and event camera processing is provided. The system acquires an output of image sensor circuitry and determines a first object classification result based on feature points associated with first objects in a first frame of the acquired output. The system executes an event-camera signal processing operation on at least two frames of the acquired output to generate an event frame. The generated event frame includes pixel-level movement information associated with second objects. The system determines a second object classification result based on the pixel-level movement information and determines one or more object classification tags based on the determined first and the determined second object classification result. The determined one or more object classification tags correspond to one or more objects, included in at least the first objects and the second objects.