Temporal Image Upscaling With Motion-Guided Texture Stabilization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image upscaling techniques using machine learning models often result in inconsistent texture across frames, leading to visible flicker and impairing user experience in video playback.
Innovation Solution
A system utilizing multiple machine learning models in parallel branches for spatial and temporal processing, including motion vector generation, blending weights calculation, and image warping to stabilize textures across frames.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If super resolution upscaling is used to reconstruct high-resolution images from low-resolution images, then image resolution is improved, but texture consistency deteriorates causing visible flicker
Solution Approach 1:
The system segments the upscaling process into two distinct branches: a spatial branch that performs super resolution upscaling on individual frames, and a temporal branch that performs motion compensation and frame blending. This segmentation allows each branch to specialize in its function, resolving the contradiction by separating resolution enhancement from texture stabilization.
Solution Approach 2:
The system changes the parameter of texture consistency by introducing temporal blending weights that dynamically adjust the contribution of previous frames. By modifying the blending ratio based on motion vectors and frame comparison, the system maintains stable textures across frames while still achieving high-resolution output.
2Stability of the object's composition
If multiple machine learning models are used for spatial and temporal processing, then texture consistency is improved, but device complexity increases
Solution Approach 1:
The processing system is segmented into parallel spatial and temporal branches, each with dedicated machine learning models. The spatial branch handles upscaling while the temporal branch handles motion compensation and blending. This segmentation organizes complexity into manageable, independent modules that can be processed simultaneously.
Solution Approach 2:
The system adds a temporal dimension to the traditional spatial upscaling process. By introducing time as a new dimension with previous frames as input, the system achieves texture stabilization without significantly increasing spatial complexity, as the temporal processing operates on already-upscaled frames through blending operations.
Data Source
AI summary
Techniques and systems are provided for image processing. For instance, a process can include generating motion vectors representing differences between a past image of a video and a current image of a video, wherein the current image occurs after the past image in the video, wherein the motion vectors are generated using a first machine learning model; applying the motion vectors to a third image to generate a warped third image; generating blending weights based on a comparison between the warped third image and the current image, wherein the blending weights are generated using a second machine learning model; upscaling the current image to generate an upscaled current image, wherein the upscaled current image is upscaled by a third machine learning model; combining the upscaled current image and a warped fourth image using the blending weights to generate an upscaled output image; and outputting the upscaled output image.