Deconvolution Layer Partitioning for Neural Network Upscaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current image upscaling technologies, such as FSRCNN, require excessive computation for upscaling FHD images to UHD, leading to large hardware requirements, high manufacturing costs, and difficulties in implementing high-speed image processing on user terminals due to significant computation bottlenecks at the deconvolution layer.
Innovation Solution
The proposed solution involves forming multiple deconvolution layers with partition layers and a remapping layer to reduce computation by distributing operations across multiple layers, allowing for a smaller, lower-cost hardware implementation and improved operating speed through the pipeline technique.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single deconvolution layer is used for image upscaling, then the structure is simple, but the computation amount is extremely large and picture quality degrades
Solution Approach 1:
The patent divides a single deconvolution layer into multiple deconvolution layers (first deconvolution layer, second deconvolution layer, etc.). Each layer performs partial upscaling operations with smaller filter sizes (e.g., 3x3 or 5x5 filters), distributing the computational load across multiple stages rather than requiring one large computation-intensive layer, thereby reducing the computation amount while maintaining structural organization
2Area of stationary object
If a single deconvolution layer is used for image upscaling, then the hardware size can be small, but picture quality degrades and computation bottlenecks occur
Solution Approach 1:
The patent divides a single deconvolution layer into multiple deconvolution layers (first deconvolution layer, second deconvolution layer, etc.). Each layer performs partial upscaling operations with smaller filter sizes (e.g., 3x3 or 5x5 filters), distributing the computational load across multiple stages rather than requiring one large computation-intensive layer, thereby reducing the computation amount while maintaining structural organization
Solution Approach 2:
The patent introduces a temporal dimension by processing upscaling in multiple sequential layers rather than a single spatial operation. This multi-layer approach allows intermediate feature maps to be generated and processed step-by-step, improving picture quality through progressive refinement while keeping individual layer hardware requirements manageable
3Productivity
If multiple deconvolution layers are used, then computation amount is reduced and hardware size decreases, but the difference in computation between layers may increase
Solution Approach 1:
The patent assigns different filter sizes to different deconvolution layers based on their specific functions. The first deconvolution layer uses smaller filters (e.g., 3x3) for initial upscaling, while subsequent layers may use progressively larger filters (e.g., 5x5) for finer detail restoration. This local optimization allows each layer to be computationally efficient for its specific purpose while collectively achieving high-quality upscaling
Data Source
AI summary
Disclosed is an image upscaling apparatus that includes: multiple convolution layers, each configured to receive an input image or a feature map outputted by a previous convolution layer and extract features to output a feature map; and a multilayer configured to receive a final feature map outputted from the last convolution layer and output an upscaled output image. The multilayer includes: a first partition layer including first filters having a minimum size along the x-axis and y-axis directions and the same size as the final feature map along the z-axis direction; and at least one second partition layer, each including second filters, having a size greater than that of the first filter in the x-axis and y-axis directions and having a number and size of the first filter in the z-axis direction, and configured to shuffle features in the x-axis and y-axis directions of the first shuffle map.