High-Resolution Frame Rate Enhancement via Low-Resolution Interpolation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video processing technologies face challenges in maintaining high frame rates for high-resolution video without introducing visual artifacts, particularly in virtual reality applications where low frame rates can cause motion judder and discomfort, and increasing bitrates to achieve this is impractical due to bandwidth constraints.
Innovation Solution
The technique involves obtaining a series of high-resolution and low-resolution frames, interpolating or extrapolating new high-resolution frames based on existing ones, down-sampling, determining weights by comparing blocks, and generating new high-resolution frames through weighted averages to increase frame rate while minimizing visual artifacts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If high frame rate bitstreams are used to achieve high frame rate, then frame rate is improved, but data size becomes very large
Solution Approach 1:
The video stream is segmented into alternating high-resolution frames and low-resolution frames. The high-resolution frames provide detailed information while the low-resolution frames provide temporal coverage. This segmentation allows the system to achieve high frame rates without transmitting full high-resolution data for every frame, thus reducing overall data size while maintaining perceived quality.
Solution Approach 2:
The resolution parameter is dynamically changed based on frame type. High-resolution frames are transmitted at full resolution, while intermediate frames are transmitted at low resolution. The decoder then synthesizes high-resolution intermediate frames by combining motion compensation from high-resolution frames with detail information from low-resolution frames, effectively changing the resolution parameter selectively to reduce data size.
2Speed
If high frame rate bitstreams are used to achieve high frame rate, then frame rate is improved, but decoding complexity increases
Solution Approach 1:
The decoding process is segmented into different stages: decoding high-resolution frames, decoding low-resolution frames, and synthesizing intermediate high-resolution frames. This segmentation allows the use of simplified interpolation techniques for intermediate frames rather than full decoding, reducing overall decoding complexity while maintaining high frame rate output.
Solution Approach 2:
Instead of fully decoding every high-resolution frame, the system copies and interpolates information from previously decoded high-resolution frames to generate intermediate high-resolution frames. This copying approach significantly reduces decoding complexity for intermediate frames while maintaining visual quality through motion compensation and detail transfer.
3Speed
If low-resolution frames are used to fill gaps, then frame rate is improved, but visual quality may deteriorate
Solution Approach 1:
Different quality levels are applied to different temporal positions. High-resolution quality is maintained at keyframes and intermediate frames are synthesized with high quality using local motion compensation and detail transfer from neighboring high-resolution frames. This local quality approach ensures that each region of the image is reconstructed with appropriate detail based on local motion characteristics.
Solution Approach 2:
Low-resolution frames act as intermediaries that guide the synthesis of high-resolution intermediate frames. The low-resolution frames provide temporal positioning and motion information, while high-resolution detail information is transferred from neighboring high-resolution frames. This intermediary approach allows the system to achieve high frame rates with high visual quality by using low-resolution frames as a scaffold for high-resolution reconstruction.
Data Source
AI summary
A device obtains a plurality of original high-resolution frames and a low-resolution frame. The device generates, based on the plurality of original high-resolution frames, a first additional high-resolution frame. The first additional high-resolution frame and the low-resolution frame correspond to a same output time. The device generates a down-sampled frame by down-sampling the first additional higher-resolution frame. Additionally, the device determines, based on comparisons of blocks of the low-resolution frame and blocks of the down-sampled frame, a plurality of weights. The device generates an up-sampled frame by up-sampling the low-resolution frame. The device generates a second additional high-resolution frame based on a weighted average of the up-sampled frame and the first additional high-resolution frame. The weighted average is based on the plurality of weights. The second additional high-resolution frame corresponds to the same output time as the first additional high-resolution frame and the low-resolution frame.