Hybrid Event-Frame Image Processing Synchronization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
There is a lack of effective utilization methods for combining event-driven type vision sensors with frame type vision sensors, limiting their joint operational capabilities.
Innovation Solution
The proposed solution involves an image processing apparatus that synchronizes timestamps between an RGB camera (frame type vision sensor) and an Event Driven Sensor (EDS) (event-driven type vision sensor), allowing for calibration and matching of images with event signals. This system includes a movement estimation unit, an inverse filter generation unit, and a filter application unit to reduce image blur caused by target movement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If event-driven type vision sensor is used, then processing speed and power consumption are improved, but image completeness and temporal resolution are worsened
Solution Approach 1:
The patent combines event-driven vision sensor and frame-type vision sensor into a hybrid system. The event-driven sensor provides high-speed change detection while the frame-type sensor provides complete temporal information. By merging both sensor types and synchronizing their timestamps, the system achieves both high processing speed and complete temporal resolution.
Solution Approach 2:
The patent introduces a synchronization mechanism as an intermediary that aligns the timestamps from both event-driven and frame-type sensors. This intermediary component enables the two different sensor types to work together seamlessly, resolving the contradiction between their different operating characteristics.
2Use of energy by stationary object
If event-driven type vision sensor is used, then power consumption is reduced, but image quality and detail are worsened
Solution Approach 1:
The system merges event-driven sensor (low power, high speed) with frame-type sensor (complete image data). The event-driven sensor triggers processing only when changes occur, reducing power consumption, while the frame-type sensor provides complete image quality. Both sensors work together to compensate for each other's weaknesses.
3Loss of time
If frame type vision sensor is used, then image completeness is improved, but processing speed and power consumption are worsened
Solution Approach 1:
The patent segments the vision system into two independent parts: event-driven sensor for change detection and frame-type sensor for complete imaging. Each sensor handles specific tasks according to its strengths, allowing the system to achieve both high processing speed and complete temporal information without relying on a single sensor type.
4Productivity
If combination of event-driven and frame type sensors is implemented, then processing capability is improved, but system complexity is worsened
Solution Approach 1:
The patent creates a universal synchronization framework that handles multiple sensor types through a common timestamp-based coordination mechanism. This multi-functional approach allows the system to process data from both event-driven and frame-type sensors using unified processing logic, reducing the actual complexity increase despite having multiple sensor types.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An image processing apparatus includes a movement estimation unit that estimates, on the basis of a first timestamp provided to an image captured by synchronous scanning, an event signal generated corresponding to an intensity change of light at one or a plurality of pixels of the image, and a second timestamp that is provided to the event signal and is in synchronism with the first timestamp, a movement of an imaging target in the image, an inverse filter generation unit that generates an inverse filter on the basis of the movement, and a filter application unit that applies the inverse filter to the image.