Neural Network Residual Block Selection for Image Upscaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image upscaling technologies, such as NVIDIA's DLSS, are resource-intensive and not suitable for resource-constrained devices like mobile SoCs due to their complex neural network structures, limiting their ability to upscale low-resolution images efficiently on these platforms.
Innovation Solution
An electronic device with a first neural network and a second neural network, including residual blocks and an upscaling block, selectively enables data propagation through convolution layers of selected residual blocks to efficiently upscale images, using control networks to determine the optimal residual blocks for each input patch image based on fidelity scores.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If complex neural network structures with many layers and filters are used for image upscaling, then image upscaling quality is improved, but computation amount increases and power consumption increases
Solution Approach 1:
The neural network is divided into multiple residual blocks, each processing a specific frequency band or feature type. The system segments the image processing task into parallel operations that can be selectively executed, reducing overall computation while maintaining quality.
Solution Approach 2:
The system dynamically adjusts the number and type of residual blocks executed based on input image characteristics, device performance, and power constraints. This allows the network to adapt its complexity in real-time, using more blocks for quality-critical tasks and fewer blocks when power is constrained.
2Manufacturing precision
If complex neural network structures with many layers and filters are used for image upscaling, then image upscaling quality is improved, but device complexity increases
Solution Approach 1:
The complex neural network is segmented into multiple independent residual blocks with standardized structures. Each block handles a specific processing function, making the overall complex system manageable through modular decomposition and enabling selective execution of individual blocks.
Solution Approach 2:
The system changes structural parameters such as the number of residual blocks, filter sizes, and activation functions based on input image characteristics and device capabilities. This allows the same base architecture to operate at different complexity levels while maintaining effective upscaling.
3Manufacturing precision
If all residual blocks are executed for every input image, then image upscaling quality is maintained, but computation time increases
Solution Approach 1:
The system dynamically determines which residual blocks to execute based on real-time analysis of input image characteristics, device performance metrics, and power availability. This dynamic selection optimizes the balance between processing speed and output quality for each specific processing task.
Solution Approach 2:
Instead of always executing all residual blocks, the system performs partial action by selectively running only the necessary subset of blocks required to achieve acceptable quality for the current input. This reduces computation time while maintaining sufficient upscaling quality for the specific use case.
Data Source
Figure 1~2
Figure 3
Figure 4A~4B
AI summary
An electronic device includes a memory for storing a first neural network and a second neural network including a plurality of residual blocks and an upscaling block, and a processor for selecting a residual block from among the plurality of residual blocks for an input patch image of a first frame based on the second neural network and generating an output patch image of the first frame by upscaling the input patch image of the first frame to an image having a target resolution based on the selected residual block and the upscaling block.