Neural Network Image Upscaling for High Frame Rate Stability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generating high-resolution image and video content is resource-intensive, particularly for devices with limited capacity, and achieving target frame rates is challenging, often constraining the quality of the resulting content with artifacts like ghosting and lag.
Innovation Solution
A renderer generates low-resolution images, which are then upscaled using a neural network-based upscaler that leverages sub-pixel jitter and blending algorithms to produce high-resolution images with temporal stability, utilizing deep learning for efficient reconstruction and temporal smoothing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If high-resolution rendering is performed directly, then image quality is improved, but processing resources and time consumption increase significantly
Solution Approach 1:
The rendering process is segmented into two stages: first rendering at low resolution, then using a neural network upscaler to generate high-resolution output. This divides the complex high-resolution rendering task into manageable parts that can be processed more efficiently.
Solution Approach 2:
The neural network upscaler is pre-trained on high-resolution image data to learn patterns and details. During runtime, this pre-trained model quickly generates high-resolution images from low-resolution inputs without requiring computationally intensive real-time high-resolution rendering.
2Productivity
If low-resolution rendering is used to reduce resource consumption, then processing speed is improved, but image quality deteriorates
Solution Approach 1:
The neural network upscaler acts as an intermediary between low-resolution rendering output and the final high-resolution display. It takes the low-resolution image as input and transforms it into a high-resolution image that maintains visual quality without requiring the original high-resolution rendering process.
Solution Approach 2:
The system changes the resolution parameter dynamically: rendering at low resolution for efficiency, then using the neural network to transform the resolution parameter while preserving or enhancing image quality through learned patterns and temporal blending.
3Speed
If frame rate is increased to meet target requirements, then temporal stability is improved, but processing resources exceed device capacity
Solution Approach 1:
The frame generation process is segmented so that only the upscaling and blending operations need to run at full frame rate, while the base rendering can operate at lower rates. This allows meeting frame rate requirements with reduced overall processing demands.
Solution Approach 2:
Instead of fully rendering all frames at high resolution, the system performs partial high-resolution processing only where needed (through the neural network upscaler), applying enhancement selectively to achieve the appearance of high frame rate performance with reduced resource consumption.
4Manufacturing precision
If neural network upscaling is applied, then image quality is improved, but device resource requirements increase
Solution Approach 1:
Instead of requiring expensive high-resolution rendering hardware, the system uses a trained neural network model (a software copy of rendering knowledge) to generate high-resolution output from low-resolution inputs. This copies the quality benefits of high-resolution rendering without the computational cost.
Solution Approach 2:
The system uses computationally inexpensive low-resolution rendering as a base, accepting that individual frames are lower quality, but relies on the neural network to quickly generate high-quality output, effectively discarding the need for expensive per-frame high-resolution processing.
Data Source
AI summary
Apparatuses, systems, and techniques are presented to generate images. In at least one embodiment, one or more neural networks are used to generate one or more images using one or more pixel weights determined based, at least in part, on one or more sub-pixel offset values.