Multi-Dimensional Tensor Permutation for NPU Cache Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Specialized processor hardware designs, such as neural processing units and graphics processing units, face difficulties in efficiently performing operations that require permutation of multi-dimensional tensor dimensions, which is necessary for certain neural networks and data processing tasks like super-resolution image processing and improved cache access.
Innovation Solution
A method and processor architecture that transfers tensor values from a first storage unit to a second storage unit by reading along a first dimension and writing to a second dimension, utilizing programmable engines and DMA engines to permute dimensions efficiently, even when storage capacity is limited.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If specialized processor hardware designs (NPU/GPU) are used, then certain operations can be performed efficiently in parallel, but operations requiring permutation of multi-dimensional tensor dimensions become difficult to perform efficiently
Solution Approach 1:
The dimension permutation operation is divided into multiple smaller operations. The tensor is processed in blocks, and each block undergoes independent dimension reordering. This segmentation allows the specialized processor to handle the permutation task through multiple simple parallel operations rather than one complex operation, resolving the contradiction between parallel processing efficiency and ease of performing dimension permutation.
Solution Approach 2:
The patent introduces an intermediate storage dimension to facilitate the permutation operation. By temporarily storing tensor data in an intermediate buffer arranged according to the target dimension layout, the system transforms the difficult in-place permutation into a series of simpler read-write operations across different memory dimensions, making the operation feasible on specialized hardware.
2Productivity
If multi-dimensional tensor dimension permutation is performed, then cache access patterns are improved and processing efficiency increases, but the complexity of the processor hardware increases
Solution Approach 1:
An intermediate storage buffer is introduced as a mediator between the input tensor and the permuted output. This intermediate buffer is arranged in the target dimension layout, allowing the system to read from the input tensor and write to the buffer without requiring complex in-place permutation logic in the processor hardware. The intermediary buffer absorbs the complexity of dimension reordering, keeping the processor design simpler while achieving efficient processing.
3Ease of operation
If tensor values are transferred between storage units by reading along one dimension and writing along another, then dimension permutation is achieved, but memory bandwidth consumption increases
Solution Approach 1:
The tensor is segmented into smaller blocks that are processed and transferred in parallel. By dividing the large tensor into multiple smaller sub-tensors, the system can perform dimension permutation on smaller data chunks simultaneously, reducing the total memory bandwidth required compared to transferring the entire tensor at once. The segmentation allows parallel memory operations that lower overall bandwidth consumption.
Data Source
AI summary
A method performed by a processor for permuting dimensions of a multi-dimensional tensor is described. The multi-dimensional tensor contains an array of tensor values in three or more dimensions that are stored in a first storage unit. The array of tensor values is transferred from the first storage unit to a second storage unit by reading tensor values from the first storage that are arrayed along a first dimension of the multi-dimensional tensor and writing the corresponding tensor values to the second storage in locations corresponding to a second dimension of the multi-dimensional tensor. The dimensions of the multi-dimensional tensor may be further permuted by a programmable engine within the processor.