Neural Network Upsampling via Pixel Copying and Pooling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Bilinear upsampling operations in neural networks are inefficiently supported by general neural network accelerators, requiring additional processing or cores due to high computational complexity and variable pixel references, which can lead to inefficient processing.
Innovation Solution
A neural network operation apparatus and method that generates an upsampled tensor by copying pixels based on a scale factor and performs a pooling operation, including average pooling, to efficiently achieve bilinear upsampling using existing hardware optimized for average pooling operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If bilinear upsampling is processed through offloading or additional core, then bilinear upsampling can be performed, but processing efficiency deteriorates due to large load compared to overall computation
Solution Approach 1:
The patent segments the bilinear upsampling operation into two distinct steps: nearest-neighbor upsampling followed by average pooling. This segmentation allows each step to be handled by standard neural network accelerator operations, avoiding the need for specialized bilinear upsampling hardware or additional processing cores, thereby maintaining high processing efficiency while achieving bilinear upsampling functionality.
2Adaptability or versatility
If bilinear upsampling is performed with core capable of bilinear upsampling, then upsampling can be executed, but processing becomes inefficient due to large computational complexity from variable pixel references
Solution Approach 1:
The patent divides the complex bilinear upsampling computation into simpler operations: first performing nearest-neighbor upsampling with fixed pixel references, then applying average pooling. This segmentation reduces computational complexity by replacing variable pixel reference lookups with standard convolutional pooling operations that have fixed kernels and strides, making the process more efficient on general neural network accelerators.
Solution Approach 2:
The patent introduces an intermediate representation by using nearest-neighbor upsampling as a bridge between the input feature map and the final bilinear upsampled output. This intermediary step transforms the problem into a sequence of simpler operations (nearest-neighbor copy followed by average pooling) rather than directly computing bilinear upsampling, thereby reducing computational complexity.
3Device complexity
If general neural network accelerator is used without bilinear upsampling support, then hardware remains simple, but bilinear upsampling cannot be performed efficiently
Solution Approach 1:
The patent segments bilinear upsampling into nearest-neighbor upsampling and average pooling operations, both of which can be efficiently performed by standard neural network accelerators without specialized bilinear upsampling hardware. This allows general-purpose hardware to achieve bilinear upsampling functionality by composing simpler operations, maintaining hardware simplicity while improving processing capability.
Solution Approach 2:
The patent makes general neural network accelerator operations universal by showing that bilinear upsampling can be achieved through combinations of standard operations (nearest-neighbor upsampling and average pooling). This multi-functionality allows the same hardware to perform both standard neural network operations and bilinear upsampling without requiring specialized dedicated hardware, thereby improving productivity on existing hardware.
Data Source
AI summary
Neural network operation apparatus and method are provided. The neural network operation apparatus includes: one or more processors; and memory storing instructions configured to cause the one or more processors to: generate an upsampled tensor by copying pixels, of a unit of data, based on a scale factor for upsampling; and generate, based on the scale factor, a neural network operation result by performing a pooling operation on the upsampled tensor.