Pixel Shuffle Neural Network Layout for Mobile Image Upsampling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning models with U-Net structures require significant computational resources and result in decreased performance due to ineffective interpolation or deconvolution operations with up-sampling, making real-time image processing on mobile devices challenging.
Innovation Solution
Implementing a neural network model with pixel unshuffle and shuffle layers, where unshuffling and shuffling operations are performed by different processors, allowing for parallel processing and reducing computational load on GPUs or DSPs, and using a heterogeneous processor division to drive lightweight models on mobile devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If interpolation or deconvolution operations with up-sampling are used to increase image size, then the desired image processing may be implemented, but computational capabilities are increased and performance is decreased
Solution Approach 1:
The image processing task is segmented into multiple stages: down-sampling the input image to reduce computational load, performing convolution operations on the smaller image, and then up-sampling the result. This segmentation allows the system to achieve desired image processing quality while maintaining computational efficiency by avoiding direct convolution on large images.
Solution Approach 2:
The patent transforms the problem from operating directly on high-resolution images to operating on down-sampled images in a different dimensional scale. By changing the resolution dimension and performing operations in this transformed space, the system achieves comparable processing quality with significantly reduced computational requirements.
2Productivity
If a lightweight model is used for deep learning on mobile devices, then computational resources are reduced, but performance may be decreased compared to heavier models
Solution Approach 1:
The patent changes key parameters of the neural network model by replacing standard convolutional layers with a hybrid architecture that combines down-sampling, convolution on reduced-size images, and up-sampling operations. This parameter transformation allows the model to maintain high processing performance while significantly reducing computational resource requirements for mobile deployment.
3Measurement precision
If convolution operations are performed on full-resolution images, then processing accuracy is maintained, but memory access and computational load increase significantly
Solution Approach 1:
The processing pipeline is segmented into distinct resolution stages: an initial down-sampling stage that reduces image dimensions before convolution operations, followed by convolution on the smaller image, and a final up-sampling stage. This segmentation enables the system to perform accurate processing on the down-sampled representation while reconstructing the final high-resolution output, thereby reducing memory access and computational load.
Solution Approach 2:
The down-sampling operation is performed preliminarily before the main convolution processing. By pre-reducing the image size, the system prepares the data in a form that requires fewer computational resources for subsequent operations, while the final up-sampling restores the necessary output resolution, thus reducing overall energy consumption without sacrificing final output quality.
Data Source
AI summary
A computing device and an operation method thereof are disclosed. The method includes unshuffling first image data to generate input data, generating output data by implementing a neural network (NN) model provided the input data, and generating second image data by shuffling the output data.