Frame Upsampling with Jitter Alignment and Residual Refinement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing super resolution techniques face challenges in achieving high-quality, low-latency image upsampling, particularly in resource-constrained devices, due to the computational demands of neural networks and inefficiencies in conventional processing methods.
Innovation Solution
A method and system that utilize a jitter pattern across a sequence of frames to align input pixel values with upsampled locations, combined with neural networks for refinement, to determine upsampled pixel values efficiently, reducing computational complexity and resource requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If neural networks are used for upsampling, then image quality is improved, but processing time and power consumption increase
Solution Approach 1:
The patent segments the upsampling process into two distinct stages: a fast initial upsampling stage using conventional interpolation methods, and a refinement stage using neural networks only on the difference/residual values. This segmentation allows the majority of pixels to be processed quickly while applying computationally intensive neural network operations only where needed, thereby reducing overall processing time while maintaining image quality.
Solution Approach 2:
The patent applies neural network processing partially - only to residual error values rather than all pixel values. By identifying that full neural network processing of every pixel is excessive, the invention applies the computationally intensive operation only to the difference between predicted and actual values, achieving high-quality results with significantly reduced computational burden.
2Manufacturing precision
If neural networks are used for upsampling, then image quality is improved, but power consumption increases
Solution Approach 1:
The patent segments the upsampling process into two distinct stages: a fast initial upsampling stage using conventional interpolation methods, and a refinement stage using neural networks only on the difference/residual values. This segmentation allows the majority of pixels to be processed quickly while applying computationally intensive neural network operations only where needed, thereby reducing overall processing time while maintaining image quality.
Solution Approach 2:
The patent applies neural network processing partially - only to residual error values rather than all pixel values. By identifying that full neural network processing of every pixel is excessive, the invention applies the computationally intensive operation only to the difference between predicted and actual values, achieving high-quality results with significantly reduced computational burden.
3Productivity
If conventional upsampling methods are used, then processing speed is improved, but image quality deteriorates
Solution Approach 1:
The patent segments the upsampling process into two distinct stages: a fast initial upsampling stage using conventional interpolation methods, and a refinement stage using neural networks only on the difference/residual values. This segmentation allows the majority of pixels to be processed quickly while applying computationally intensive neural network operations only where needed, thereby reducing overall processing time while maintaining image quality.
Solution Approach 2:
The patent introduces an intermediary element - the residual error values - that mediates between the fast conventional upsampling method and the high-quality neural network method. By computing the difference between the initial upsampled image and the target high-resolution image, the invention creates a refined input for neural network processing that bridges the gap between speed and quality requirements.
4Manufacturing precision
If high-resolution rendering is performed directly, then image quality is improved, but computational resources and cost increase
Solution Approach 1:
The patent performs preliminary upsampling using fast conventional methods before applying neural network refinement. By pre-processing the low-resolution image to generate an initial high-resolution version, the invention reduces the complexity of the subsequent neural network processing, as the network only needs to refine differences rather than generate all pixel values from scratch.
Solution Approach 2:
The patent applies neural network processing partially - only to residual error values rather than all pixel values. By identifying that full neural network processing of every pixel is excessive, the invention applies the computationally intensive operation only to the difference between predicted and actual values, achieving high-quality results with significantly reduced computational burden.
Data Source
AI summary
A method and processing system for applying upsampling to input pixel values of frames of a sequence of frames to determine upsampled pixel values at upsampled pixel locations. A jitter pattern is used over the sequence such that different frames of the sequence have input pixel values at locations corresponding to different upsampled pixel locations. An initial block of upsampled pixel values is determined for a current frame. An aligned block of upsampled pixel values for the current frame is determined based on the initial block in accordance with the jitter pattern. A block of refinement values for the initial block of upsampled pixel values is determined for the current frame, and is applied to the initial block to determine a refined block of upsampled pixel values.