Neural Video Frame Interpolation for FPS Mismatch and Motion Blur
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The mismatch in frames per second (FPS) between image data and the apparatus reproducing it leads to deteriorated image quality, necessitating frame interpolation or synthesis to improve video quality.
Innovation Solution
A display device employs neural networks for motion estimation and object detection to generate interpolation frames by using pre-generated mapping tables and metadata, reducing data processing complexity and increasing FPS through frame rate conversion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If conventional video processing methods are used, then device complexity is reduced, but video quality deteriorates due to block artifacts and blurriness
Solution Approach 1:
The video processing method segments the video data into multiple frames and further divides each frame into blocks, then applies different processing operations to different blocks based on their characteristics. This segmentation allows selective processing that improves video quality while managing computational complexity through localized operations rather than global processing.
Solution Approach 2:
The patent applies different processing operations to different blocks within video frames based on local characteristics such as motion magnitude and block type (intra-block vs inter-block). Motion-compensated interpolation is applied to blocks with high motion magnitude, while simpler operations are used for static blocks, optimizing the balance between quality improvement and processing complexity.
2Manufacturing precision
If motion-compensated interpolation is applied to all blocks, then video quality improves, but processing time and complexity increase significantly
Solution Approach 1:
Instead of applying motion-compensated interpolation to all blocks, the patent selectively applies this operation only to blocks with high motion magnitude that would benefit most from it. Static or low-motion blocks receive simpler processing, reducing overall processing time while maintaining video quality where it matters most.
Solution Approach 2:
The patent changes the processing parameter (interpolation operation type) based on the motion magnitude parameter of each block. By dynamically adjusting the processing intensity according to local motion characteristics, the system achieves high interpolation accuracy where needed while minimizing processing time for static regions.
3Manufacturing precision
If advanced processing operations are used, then video quality improves, but ease of operation deteriorates due to complex processing requirements
Solution Approach 1:
The video processing system automatically analyzes each block's characteristics (motion magnitude, block type) and selects the appropriate processing operation without requiring manual intervention. The system self-adjusts the processing complexity based on the content being processed, making advanced video quality enhancement accessible without requiring operator expertise in complex processing parameters.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Provided is a method of improving video quality. The method of improving video quality includes obtaining first motion information regarding a user control object from a frame included in a video, obtaining second motion information regarding remaining pixels excluding pixels included in the user control object from pixels included in the frame, by using a first neural network, and generating an interpolation frame between at least two frames from the at least two frames, the first motion information, and the second motion information, by using a second neural network.