Systolic Array Decomposition for Irregular Tensor Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Systolic array accelerators face inefficiency when handling irregularly shaped tensors, as they underutilize the available processing elements, leading to increased clock cycles and reduced computational throughput.
Innovation Solution
The systolic array is decomposed into smaller N x N sub-arrays, and input tensors are apportioned into N x M and M x N arrays, allowing for simultaneous mathematical operations across multiple sub-arrays to maximize cell utilization and reduce clock cycles.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the systolic array is used with irregularly shaped tensors (N x M and M x N), then the system can handle diverse tensor operations, but the processing efficiency decreases due to underutilization of processing elements
Solution Approach 1:
The patent divides the systolic array into multiple N x N sub-arrays (e.g., four 2x2 sub-arrays within a 4x4 array). Each sub-array independently processes a portion of the input tensors, allowing irregularly shaped tensors to be handled efficiently by distributing work across multiple specialized units rather than using the entire array for each operation.
Solution Approach 2:
The patent introduces a new dimension of parallelism by organizing processing elements into a two-dimensional grid of N x N sub-arrays. This allows simultaneous processing of multiple tensor operations across different spatial locations, transforming the single-stream processing of irregular tensors into multi-stream parallel processing that fully utilizes available hardware resources.
2Adaptability or versatility
If the systolic array processes irregularly shaped tensors, then various tensor operations can be performed, but the number of clock cycles increases
Solution Approach 1:
By segmenting the systolic array into multiple N x N sub-arrays, the patent enables parallel processing of tensor operations. Each sub-array processes a specific portion of the input tensors simultaneously, reducing the total number of clock cycles required compared to sequential processing by a single array.
Solution Approach 2:
The patent combines multiple N x N sub-arrays into a unified systolic array structure that processes different portions of the same tensor operation in parallel. The results from each sub-array are merged to produce the final output, achieving both versatility for irregular tensor shapes and reduced execution time through parallelism.
3Adaptability or versatility
If the systolic array cells are not fully utilized, then the system can handle irregular tensor shapes, but computational efficiency decreases
Solution Approach 1:
The patent segments the systolic array into multiple N x N sub-arrays that can be independently activated. This segmentation allows the system to precisely match the computational workload to the available processing elements, ensuring that each active cell performs useful work and minimizing energy waste from idle cells.
Solution Approach 2:
Different regions of the systolic array (different N x N sub-arrays) are assigned to process different portions of the input tensors. This local specialization ensures that each processing element operates at full capacity on its assigned data, maximizing computational efficiency while maintaining the ability to handle irregular tensor shapes through appropriate data distribution.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The present disclosure is directed to systems and methods for decomposing systolic array circuitry to provide a plurality of N x N systolic sub-array circuits, apportioning a first tensor or array into a plurality of N x M first input arrays, and apportioning a second tensor or array into a plurality of M x N second input arrays. Systolic array control circuitry transfers corresponding ones of the first input arrays and second input arrays to a respective one of the plurality of N x N systolic sub-array circuits. As the elements included in the first input array and the elements included in the second input array are transferred to the systolic sub-array, the systolic sub-array performs one or more mathematical operations using the first and the second input arrays. The systems and methods beneficially improve the usage of the systolic array circuitry thereby advantageously reducing the number of clock cycles needed to perform a given number of calculations.