Neural Network Image Upsampling with Blending Weights
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generating high-quality image and video content at higher resolutions is resource-intensive, particularly for devices with limited capacity, and determining optimal blending weights for temporal smoothing is challenging, often resulting in noisy or artifact-ridden images, especially in dynamic scenes.
Innovation Solution
A system that uses a neural network to upscale images from a lower resolution to a higher resolution, incorporating jitter-aware upsampling and blending weights determined by a clamping mask and variance mask, which helps in reducing noise and artifacts by aligning jitter offsets and per-frame samples with a history buffer, and applying appropriate filtering to achieve temporal convergence.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Length of stationary object
If image upsampling is performed using traditional methods, then resolution is increased, but image quality deteriorates with noise and artifacts
Solution Approach 1:
The system performs preliminary actions by collecting multiple low-resolution frames and performing downsampling operations before upsampling. The history buffer stores processed frame data that is later combined with current frame data during upsampling, preparing the data in advance to reduce noise and artifacts in the final high-resolution output.
Solution Approach 2:
The system implements feedback by using a history buffer that stores previously processed frame data. This historical information is fed back into the upsampling process and combined with current frame data using blending weights, allowing the system to learn from past frames and improve the quality of the current high-resolution output.
2Manufacturing precision
If temporal blending is applied to smooth pixel data between frames, then image quality improves, but artifacts such as temporal lag appear in dynamic scenes
Solution Approach 1:
The system applies dynamics by using variable blending weights that are determined on a per-pixel basis rather than using a fixed blending ratio. The blending component dynamically adjusts the weight given to history buffer data versus current frame data for each pixel, allowing the system to adapt to local image characteristics and motion patterns, thereby reducing temporal lag artifacts in dynamic scenes while maintaining quality in static areas.
Solution Approach 2:
The system implements local quality by determining blending weights independently for different regions of the image. Pixels in static areas can use higher blending weights for smoother quality, while pixels in dynamic areas use lower weights to avoid temporal lag, allowing each region to be optimized according to its specific characteristics.
3Manufacturing precision
If high-quality content generation is performed at higher resolutions, then image quality improves, but resource consumption increases significantly
Solution Approach 1:
The system uses copying by working with downscaled versions of frames in the history buffer during the upsampling process. Instead of directly processing full-resolution historical frames, the system creates and processes downscaled copies, reducing the computational burden and memory requirements while still achieving high-quality output through the blending and upsampling operations.
Solution Approach 2:
The system applies segmentation by dividing the image processing into distinct stages: downsampling current and historical frames, processing at reduced resolution to determine blending weights, and then upsampling to final resolution. This segmented approach allows computationally intensive operations to be performed at lower resolutions where fewer resources are required.
Data Source
AI summary
Apparatuses, systems, and techniques are presented to reconstruct one or more images. In at least one embodiment, one or more neural networks are used to determine one or more blending weights for one or more images based, at least in part, upon one or more pixel value masks for the one or more images.