Real-Time Video Super-Resolution with Selective Keyframe Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing real-time video streaming on mobile devices faces challenges in maintaining user Quality of Experience (QoE) under bandwidth constraints due to limited computing capacity, despite advancements in adaptive streaming and super-resolution (SR) technologies.
Innovation Solution
Applying deep neural network (DNN)-based SR to a small number of pre-selected video frames, enhancing the resolution of remaining frames using cache profiles and anchor frames selected within a quality margin, and providing multiple DNN options suitable for the mobile device's computing capacity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If deep neural network-based super-resolution is applied to all video frames, then video quality is improved, but computing capacity requirements and energy consumption increase significantly
Solution Approach 1:
The patent segments the video frames into two categories: keyframes and non-keyframes. Super-resolution processing is selectively applied only to keyframes, while non-keyframes are processed using lighter computational methods. This segmentation resolves the contradiction by maintaining video quality through strategic application of DNN-based SR to keyframes while significantly reducing computing capacity requirements and energy consumption by avoiding full-frame processing.
Solution Approach 2:
The patent applies super-resolution processing partially rather than to all frames. By processing only a subset of frames (keyframes) with the computationally intensive DNN-based SR method, the system achieves sufficient video quality improvement without the excessive computing capacity requirements and energy consumption that would result from processing every frame with the same method.
2Manufacturing precision
If deep neural network-based super-resolution is applied to all video frames, then video quality is improved, but energy consumption increases significantly
Solution Approach 1:
The patent segments video frames into keyframes and non-keyframes, applying energy-intensive DNN-based super-resolution only to keyframes. This segmentation reduces energy consumption by avoiding redundant processing of non-keyframes while maintaining acceptable video quality through the strategic selection of keyframes that contribute most to overall perceived quality.
Solution Approach 2:
The patent implements partial action by processing only a fraction of video frames (keyframes) with the energy-intensive super-resolution method. This approach achieves sufficient video quality improvement without the excessive energy consumption that would result from applying the same processing to all frames, thereby resolving the contradiction between quality and energy usage.
3Manufacturing precision
If super-resolution is applied to every frame, then video quality is improved, but processing time increases
Solution Approach 1:
The patent segments video frames into keyframes and non-keyframes, applying computationally intensive super-resolution processing only to keyframes. This segmentation reduces processing time by eliminating redundant processing of non-keyframes while maintaining video quality through the strategic selection of keyframes that provide the most quality improvement per unit of processing time.
Solution Approach 2:
The patent applies super-resolution processing partially to only keyframes rather than all frames. This partial action approach achieves sufficient video quality improvement while significantly reducing processing time, as the computational burden is distributed across fewer frames, thereby resolving the contradiction between quality and processing time.
4Productivity
If adaptive streaming is used to optimize bitrate, then network bandwidth utilization is improved, but user QoE depends on available bandwidth
Solution Approach 1:
The patent extracts the super-resolution processing step from the traditional adaptive streaming pipeline and positions it between the network receiver and the video player. This extraction allows the system to receive lower-bitrate video streams through adaptive streaming (improving bandwidth utilization) while subsequently enhancing video quality through DNN-based SR processing (improving user QoE independence from bandwidth constraints).
Solution Approach 2:
The patent introduces super-resolution processing as an intermediary component between the adaptive streaming receiver and the video player. This intermediary enhances the quality of received video frames regardless of their original bitrate, thereby decoupling user QoE from direct dependence on network bandwidth while still allowing adaptive streaming to optimize bandwidth utilization.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
The present disclosure seeks to perform real-time video streaming on a mobile device toward maintaining user QoE even under bandwidth constraints while being acceptable to the lightweight computing capacity of the mobile device. To this end, the embodiments apply deep neural network-based SR to a small number of pre-selected video frames and utilize the video frames to which SR is applied to enhance the resolution of the remaining frames, wherein the pre-selected frames are chosen for SR within a preset quality margin. Additionally, the present disclosure provides an apparatus and a method for SR acceleration for real-time video streaming under the lightweight computing capacity and video-specific constraints of a mobile device, which allow a server to deliver multiple options on a deep neural network and a cache profile including SR application information and enable the mobile device to select an option suitable for its computing capacity.