Video Interpolation Using Event and Image Sensors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video interpolation methods face challenges in handling highly nonlinear motion, maintaining correct global contrast, and robustness against light and focus changes, while also being energy and memory inefficient, especially in producing high-speed color videos.
Innovation Solution
A device and method combining a synchronized image sensor and event sensor to perform warping-based and synthesis-based interpolations, enabling the computation of intermediate frames that handle nonlinear motion and light/focus changes, with low energy and memory requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If single video approaches are used for frame interpolation, then the device complexity is low, but the manufacturing precision of intermediate frames deteriorates when handling highly nonlinear motion
Solution Approach 1:
The patent combines event data from event sensors with intensity frames from image sensors to perform frame interpolation. This merging of complementary data sources allows the system to handle highly nonlinear motion effectively while maintaining reasonable device complexity, as the event sensors provide motion information without requiring complex multi-camera setups.
Solution Approach 2:
The patent uses event sensors as an intermediary to capture motion information between frames. These event sensors act as mediators that detect pixel intensity changes and provide temporal information, enabling accurate intermediate frame generation without requiring complex hardware configurations.
2Manufacturing precision
If multiple video approaches are used for frame interpolation, then the manufacturing precision of intermediate frames improves, but the use of energy increases
Solution Approach 1:
The patent extracts only the necessary motion information from the scene using event sensors, which report only when pixel intensity changes occur. This selective extraction of relevant data reduces energy consumption compared to continuously capturing full video frames at high frame rates, while still providing sufficient information for accurate frame interpolation.
Solution Approach 2:
The patent uses partial action by capturing only the minimal necessary information (event triggers) rather than complete video frames. The event sensors capture only the essential motion events needed for interpolation, reducing energy usage while maintaining intermediate frame quality.
3Manufacturing precision
If multiple video approaches are used for frame interpolation, then the manufacturing precision of intermediate frames improves, but the device complexity increases
Solution Approach 1:
The patent merges event sensors and image sensors into a unified interpolation system. This combination allows the use of complementary data from both sensor types to achieve high intermediate frame quality without requiring multiple separate video acquisition systems, thus reducing overall device complexity.
4Use of energy by moving object
If event-based methods are used for frame interpolation, then the use of energy decreases, but the manufacturing precision deteriorates when producing high-speed color videos
Solution Approach 1:
The patent combines event data with intensity frame data from image sensors to produce high-speed color videos. This merging allows the system to leverage the low energy consumption of event sensors while using the image sensor data to maintain color information and overall video quality.
Solution Approach 2:
The patent uses intensity frames from image sensors as intermediaries to complement event data. These intermediary frames provide the color and detail information needed for high-quality video output, while the event data provides the temporal structure, achieving both energy efficiency and video quality.
5Device complexity
If single video approaches are used for frame interpolation, then the device complexity is low, but the reliability deteriorates when light or camera focus changes
Solution Approach 1:
The patent uses event sensors as intermediaries that detect actual pixel intensity changes in the scene. These event triggers serve as reliable indicators of motion and change, providing robust interpolation even when lighting or focus conditions change, because they are based on actual detected changes rather than assumptions about brightness constancy.
Data Source
Figure 1
Figure 2(a)~2(b)
Figure 3
AI summary
The present disclosure relates to the field of image and video processing. In particular, the disclosure is concerned with video interpolation, for instance, for slow motion video applications. Embodiments of the invention propose accordingly a device and a method for video interpolation. The device comprise a image sensor, an event sensor and a processor. The image sensor is configured to obtain a time sequence of frames including a first frame and a second frame obtained after the first frame. The event sensor is configured to obtain a time sequence of events, wherein one or more events are obtained between the first frame and the second frame. The processor is configured to compute an intermediate frame between the first frame and the second frame, based on the first frame, the second frame, and the one or more events obtained between the first frame and the second frame.