Self-Calibrated Convolution and Late Upsampling for Real-Time Super-Resolution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional image super-resolution methods face challenges in achieving accurate real-time processing due to high computational complexity and inefficiency in mapping low-resolution to high-resolution images, particularly with large CNN models, which require extensive training settings and numerous parameters.
Innovation Solution
A system utilizing depthwise separable convolution layers and a late upsampling scheme with a self-calibrated block and pixel attention (ESC-PA) to reduce model parameters and multiply-accumulate operations, enabling efficient real-time image super-resolution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the size of SRCNNs is increased to overcome accuracy limitations, then SR accuracy is improved, but the number of parameters increases and real-time execution becomes impossible
Solution Approach 1:
The patent segments the convolution operation into two separate operations: depthwise convolution (spatial filtering) and pointwise convolution (channel mixing). This segmentation allows each operation to be optimized independently, reducing the total number of parameters while maintaining representational capacity. The depthwise convolution uses 3x3 kernels for spatial features with minimal parameters, while pointwise convolution uses 1x1 kernels for channel integration, achieving a balance between accuracy and complexity.
Solution Approach 2:
The patent changes the parameter configuration by using depthwise separable convolutions instead of standard convolutions. This parameter change reduces the number of learnable parameters significantly: a standard convolution with C_in input channels, C_out output channels, and kxk kernel size has k^2*C_in*C_out parameters, while depthwise separable convolution reduces this to k^2*C_in + C_out parameters. This parameter optimization enables larger model architectures to be trained and deployed in real-time applications.
2Reliability
If conventional convolutional layers are used, then feature extraction capability is maintained, but computational complexity and MACs are high
Solution Approach 1:
The patent segments the computationally intensive convolution operation into two lighter operations: depthwise convolution that processes spatial features independently for each channel, and pointwise convolution that mixes channels using 1x1 kernels. This segmentation reduces the computational complexity from O(k^2*C_in*C_out*H*W) to O(k^2*C_in*H*W + C_in*C_out*H*W), significantly reducing MACs while preserving feature extraction capability through the sequential application of both operations.
3Measurement precision
If upsampling is performed early in the network, then high-frequency information can be refined, but the input dimension to CNN becomes large and operations become computationally expensive
Solution Approach 1:
The patent inverts the conventional upsampling timing by performing upsampling at the end of the network rather than at the beginning. This inversion allows the CNN to first extract features from the compact low-resolution input, maintaining low computational complexity throughout the feature extraction process. The upsampling operation is then applied to the extracted feature maps, and refinement operations work on the upsampled features rather than the original large input, achieving high-frequency refinement without the burden of processing large input dimensions throughout the network.
Data Source
AI summary
A system and a method for displaying super-resolution images generated from images of lower resolution, includes processor circuitry for a combination multi-core CPU and machine learning engine configured with an input for receiving the low resolution images, a feature extraction section to extract features from the low resolution images, non-linear feature mapping section, connected to the feature extraction section, generating feature maps using a self-calibrated block with pixel attention having a plurality of Depthwise Separable Convolution (DSC) layers, a late upsampling section combines at least one DSC layer and a skip connection that upsamples the feature maps to a predetermined dimension, and a video output for displaying approximate upsampled super-resolution images that corresponds to the low resolution images.