Video Super-Resolution via Optical Flow Based Frame Rate Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional video super-resolution methods fail to effectively enhance the spatial resolution of high frame rate videos beyond 240 fps, as they do not account for the unique challenges posed by high frame rates in motion estimation and resolution conversion.
Innovation Solution
A video super-resolution method that creates image groups with different frame rates, performs motion estimation for each group, and selects the appropriate group based on estimated motion amounts for weighted averaging to generate high-resolution videos, allowing for stable feature improvement even at super-high frame rates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional video super-resolution methods are applied to high frame rate videos beyond 240 fps, then processing speed is maintained, but spatial resolution enhancement becomes ineffective
Solution Approach 1:
The patent divides the video into multiple image groups with different frame rates (e.g., 30fps, 60fps, 120fps, 240fps) and applies motion estimation and super-resolution selectively to each group. This segmentation allows the system to process high frame rate content while applying enhanced spatial resolution only where motion characteristics permit, thus resolving the contradiction between maintaining frame rate and improving spatial resolution.
2Measurement precision
If motion estimation is performed for all image groups, then accurate motion compensation is achieved, but computational complexity increases
Solution Approach 1:
The patent performs motion estimation and selects image groups based on local motion characteristics. By analyzing motion amounts in different regions and selecting only those groups with suitable motion characteristics for super-resolution, the system achieves accurate motion compensation where needed while reducing computational complexity by avoiding processing of all image groups uniformly.
3Adaptability or versatility
If a single video super-resolver is used, then device simplicity is maintained, but adaptability to different motion characteristics is reduced
Solution Approach 1:
The patent employs multiple video super-resolvers (e.g., first, second, third super-resolvers) that are selectively applied based on motion characteristics of different image groups. This dynamic selection approach allows the system to adapt to varying motion characteristics across different frame rates and regions, improving versatility while managing device complexity through intelligent selection rather than universal application of all components.
Data Source
AI summary
A video super-resolution device includes a down-sampler, video super-resolvers, and a selecting/averaging unit. The down-sampler divides an input low-resolution video into a plurality of frame rates. The video super-resolvers are video super-resolvers trained at different frame rates, and perform super-resolution of the low-resolution video. The selecting/averaging unit selects a video super-resolver according to the magnitude of an optical flow obtained as a result of video super-resolution. Specifically, in a case where the optical flow has a value smaller than 0.5 pixels, the selecting/averaging unit makes selection such as not adopting a result of video super-resolution at a high frame rate. Finally, the selecting/averaging unit obtains a mean value of the selected video super-resolution result, and outputs a final high-resolution video.