Variable Rate Shading Neural Network Image Upscaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image upscaling techniques in graphics processing are computationally intensive, particularly when applied to resource-constrained devices, and often require significant memory usage.
Innovation Solution
The use of variable rate shading (VRS) in combination with machine-learning (ML) techniques for spatial and temporal upscaling, where pixel values are rendered at different shading rates and applied to trained neural networks for upscaling, reducing computational requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional upscaling techniques are applied to restore resolution after downscaling, then image resolution is improved, but computational complexity and memory usage increase significantly
Solution Approach 1:
The patent segments the image processing pipeline into distinct stages: downscaling, processing at reduced resolution, and upscaling. By dividing the computationally intensive upscaling task into manageable segments that can be performed on resource-constrained devices, the system achieves high resolution output without overwhelming computational complexity
Solution Approach 2:
The patent introduces a temporal dimension by processing multiple frames together. Instead of upscling each frame independently (spatial dimension only), the system utilizes temporal information across successive frames to reconstruct high-resolution images, reducing the computational burden on individual frames while maintaining overall image quality
2Measurement precision
If ML-based upscaling is applied to enhance image quality, then image quality is improved, but energy consumption increases
Solution Approach 1:
The patent applies ML-based upscaling selectively rather than uniformly across all image data. By identifying regions that benefit most from ML enhancement and applying the technique only where necessary, the system achieves improved image quality while minimizing energy consumption associated with running computationally intensive ML models
Solution Approach 2:
The system performs preliminary downscaling and processing at reduced resolution before applying ML upscaling. This preliminary action reduces the amount of data that requires intensive ML processing, thereby lowering energy consumption while still achieving high-quality output through the subsequent upscaling stage
3Productivity
If downscaling is applied to reduce computational load, then processing speed is improved, but image detail is lost
Solution Approach 1:
The patent dynamically changes resolution parameters based on processing requirements and available resources. By adjusting the downscaling factor and upscaling approach according to specific image characteristics and device capabilities, the system maintains essential image details while achieving processing speeds suitable for resource-constrained devices
Solution Approach 2:
The patent introduces temporal information from adjacent frames as an intermediary to preserve image details during downscaling. By utilizing motion compensation and temporal prediction, the system recovers fine details that would otherwise be lost in the downscaling process, enabling both fast processing and high-quality output
Data Source
AI summary
Example methods, apparatuses, and/or articles of manufacture are disclosed that may be implemented, in whole or in part, using one or more computing devices to enhance a rendered image. In an implementation, a process to enhance a portion of a rendered image may be affected based, at least in part, on a shading rate applied in rendering the portion of the rendered image.