Neural Video Upsampling With Motion Warping for Artifact Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video content quality is often suboptimal for display on various devices, and existing upsampling methods experience artifacts or are of lower quality than desired, particularly for live video.
Innovation Solution
A neural network-based upsampling system that utilizes temporal anti-aliasing and super-resolution techniques, incorporating historical frame data and motion warping to enhance video frame quality, with training on annotated image pairs to optimize blending factors and kernel application for improved perception.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If traditional upsampling methods are used to improve video resolution, then video quality is improved, but artifacts appear and quality deteriorates
Solution Approach 1:
The system performs preliminary actions by processing historical frames and predicting motion vectors before the actual upsampling of the current frame. This pre-processing includes warping historical frames to align with the current frame's motion, which prepares the data in advance to avoid artifacts during the final upsampling operation. The neural network is also pre-trained on high-quality image pairs to learn optimal upsampling patterns that minimize artifact generation.
Solution Approach 2:
The patent introduces an intermediary neural network that acts as a mediator between the low-resolution input video and the high-resolution output. This neural network intermediary processes the video data through multiple layers, transforming the low-resolution frames into high-resolution frames while learning to suppress artifacts. The intermediary also incorporates motion compensation mechanisms that mediate between different frames to maintain temporal consistency and reduce artifact generation.
2Manufacturing precision
If neural network processing is applied to enhance video quality, then video sharpness and resolution are improved, but processing time increases
Solution Approach 1:
The patent segments the video processing task into multiple independent components that can be processed in parallel. The video stream is divided into individual frames, and each frame is processed independently through the neural network. This segmentation allows for parallel processing of multiple frames simultaneously, reducing overall processing time while maintaining high sharpness and quality for each frame. The segmentation also enables efficient utilization of GPU resources for neural network inference.
Solution Approach 2:
The system performs preliminary actions by pre-processing video frames before neural network inference. This includes converting frames to a standardized format, extracting key features, and preparing input data in advance. The neural network model is also pre-trained and optimized beforehand, so that during actual video processing, the inference can be performed rapidly. This preliminary preparation significantly reduces the time required for real-time high-quality video processing.
Data Source
AI summary
Apparatuses, systems, and techniques to enhance video are disclosed. In at least one embodiment, one or more neural networks are used to create a higher resolution video using upsampled frames from a lower resolution video.