Tensor Transposition Layout for Variable-Width AI Memory I/O
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AI accelerators face inefficiencies in using computing resources due to the difficulty in processing tensor data when the channel dimension size is not a multiple of K, which affects parallel data processing and hardware utilization.
Innovation Solution
A computing system that variably determines memory input/output units based on processor processing units, incorporating a transposition module to efficiently manage data flow between memory and processor, optimizing data layout and transposition to align with processor capabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If K tensor elements are stored in the vector memory to enable parallel processing of K pieces of data, then parallel processing capability is improved, but hardware utilization efficiency deteriorates when the channel dimension size is not a multiple of K
Solution Approach 1:
The patent implements dynamic adjustment of the parameter K (number of parallel data elements) based on the actual channel dimension size. The vector memory and processor can adaptively change their processing unit size to match the input data characteristics, transforming a fixed-configuration system into a dynamic one that optimizes hardware utilization for different tensor sizes while maintaining high parallel processing capability
Solution Approach 2:
The system changes the parameter K from a fixed value to a variable that can be adjusted according to the channel dimension size. By modifying this key parameter, the system achieves both high parallel processing efficiency (when K is large) and good adaptability (when K is adjusted to divide evenly into the channel dimension), resolving the contradiction between these two requirements
2Device complexity
If a fixed memory entry size is used for vector processing, then hardware implementation is simplified, but data supply efficiency deteriorates when channel dimension size is not a multiple of the entry size
Solution Approach 1:
The system performs preliminary transposition operations on tensor data to reorganize it into a format that matches the fixed memory entry size requirements. By pre-processing the data layout before it enters the vector memory, the system ensures efficient data supply without requiring complex variable-sized memory entries, thus maintaining simple hardware implementation while achieving high data supply efficiency
Solution Approach 2:
The patent introduces a transposition module as an intermediary between the general tensor input and the fixed-structure vector memory. This intermediary component transforms data into the required format, allowing the memory system to maintain its simple fixed-entry structure while still achieving efficient data supply for various channel dimension sizes through the buffering and reorganization function of the transposition module
Data Source
AI summary
A computing system and a transposition method therefor. One aspect of the present disclosure provides a computing system that variably determines an input/output unit of a memory depending on a processing unit or an output unit of a processor, and a transposition method for supporting same.