Transpose Circuitry in Programmable Compute Engine
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network accelerators face inefficiencies due to the need for transpose operations, which consume processing cycles and introduce latency by using the processing engine array for data rearrangement, especially when preparing tensors for subsequent computations.
Innovation Solution
Integrating transpose circuitry into the compute engine's datapath allows for on-the-fly tensor transposition before processing, eliminating the need for the processing engine array and reducing memory usage by avoiding intermediate transposed tensor storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Shape
If the processing engine array is used to perform transpose operations for data rearrangement, then the tensor data can be properly organized for subsequent computations, but processing cycles are consumed and latency is introduced
Solution Approach 1:
The patent divides the transpose operation into two independent segments: a dedicated transpose circuit handles data rearrangement while the processing engine array simultaneously performs computational operations on different data. This segmentation eliminates the sequential dependency where computation must wait for transpose completion, thereby reducing processing latency while achieving proper data organization.
2Shape
If the processing engine array is used to perform transpose operations, then data can be rearranged, but processing cycles are consumed that could be used for meaningful computations
Solution Approach 1:
The patent separates the transpose function from the computation function by introducing a dedicated transpose circuit. This allows the processing engine array to maintain full productivity by performing computational operations without being diverted to transpose operations, while the transpose circuit independently handles data rearrangement tasks.
Solution Approach 2:
The patent introduces an intermediate buffer memory structure that decouples the transpose operation from the computation operation. The transpose circuit writes rearranged data to the buffer while the processing engine array reads original data for computation, enabling parallel execution and eliminating resource contention.
3Shape
If transpose operations are performed using the processing engine array, then data layout can be changed, but memory bandwidth is consumed for reading and writing tensor data
Solution Approach 1:
The patent introduces an intermediate buffer memory structure that decouples the transpose operation from the computation operation. The transpose circuit writes rearranged data to the buffer while the processing engine array reads original data for computation, enabling parallel execution and eliminating resource contention.
Data Source
AI summary
A technique to execute transpose and compute operations may include retrieving a set of machine instructions from an instruction buffer of a data processor. The instruction buffer has multiple entries, and each entry stores one machine instruction. A machine instruction from the set of machine instructions is executed to transpose a submatrix of an input tensor and perform computations on column elements of the submatrix. The machine instruction combines the transpose operation with computational operations into a single machine instruction.