Systolic Array NPU Sub-Kernel Processing for Transpose Convolution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural processing units (NPUs) face increased latency and unnecessary multiply-and-accumulate (MAC) operations due to the use of general convolution operations for transpose convolution operations in artificial neural networks, particularly when generating high-resolution images from low-resolution images.
Innovation Solution
The NPU divides the transpose convolution operation into multiple sub-kernel convolutions, optimizing the process by eliminating operations on zero-padded areas and reducing MAC operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If general convolution operations are used for transpose convolution operations, then the NPU can process image enhancement tasks, but latency increases and unnecessary MAC operations are performed
Solution Approach 1:
The patent segments the feature map into multiple regions based on padding patterns, identifying zero-padded areas that can be excluded from computation. This segmentation allows the NPU to perform transpose convolution only on valid data regions, eliminating unnecessary MAC operations and reducing latency while maintaining processing capability.
2Adaptability or versatility
If general convolution operations are used for transpose convolution operations, then the NPU can handle various convolution tasks, but unnecessary MAC operations increase
Solution Approach 1:
The patent applies local quality by treating different regions of the feature map differently based on their padding characteristics. Valid regions undergo full convolution operations while zero-padded regions are identified and excluded from computation. This localized differentiation reduces unnecessary MAC operations and energy consumption while preserving the NPU's versatility in handling various convolution tasks.
Data Source
AI summary
An operating method for a neural processing unit (NPU) is disclosed. The method includes determining that a first convolution layer performs a transpose convolution operation, dividing a kernel used for the transpose convolution into multiple sub-kernels, and performing convolution operations between an input feature map and each sub-kernel. The operations are performed by a plurality of processing elements (PEs), each configured to reuse at least one of an output feature map, a sub-kernel, or the input feature map stored in a local memory. The NPU includes a systolic array structure comprising multiple structures arranged in parallel, each corresponding to values stored in local memory. The stored values are used in successive convolution operations, enhancing computational efficiency and memory reuse in transpose convolution layers.