Systolic Array Parallelism via Multiple Accumulate Busses
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Systolic arrays used in artificial neural networks face challenges in achieving high inference throughput while minimizing power consumption and latency, as increasing clock speed leads to increased power consumption and cooling requirements, limiting performance improvements.
Innovation Solution
The implementation of multiple busses within the columns and rows of a systolic array allows for increased parallelism and reduced latency, enabling the use of lower-cost, lower-power processing elements by enabling sequence skipping and parallel accumulation, which decreases the overall path length of calculations and allows for longer latency operations without increasing overall latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If clock speed is increased to achieve high inference throughput, then processing speed is improved, but power consumption increases
Solution Approach 1:
The systolic array is divided into multiple independent compute blocks, each with its own accumulate bus. This segmentation allows parallel processing of multiple data streams simultaneously, increasing throughput without requiring higher clock speeds. Each block operates independently at lower frequencies while collectively achieving high overall throughput.
Solution Approach 2:
The patent introduces parallelism along the data dimension by implementing multiple accumulate busses within each column. Instead of increasing speed along a single processing path, multiple simultaneous paths are created, allowing data to flow through parallel accumulation channels. This dimensional expansion of data flow paths increases throughput without proportionally increasing power consumption.
2Loss of time
If clock speed is increased to reduce latency, then processing speed is improved, but cooling requirements increase
Solution Approach 1:
By segmenting the accumulation function across multiple busses, the patent reduces the critical path length within each compute block. Data can be accumulated in parallel across these segmented paths, reducing the time required for each operation without requiring higher clock speeds, thereby reducing heat generation and cooling requirements.
Solution Approach 2:
The multiple accumulate busses enable continuous data flow and accumulation operations without idle waiting periods. Data streams can be processed continuously through parallel paths, reducing overall latency without requiring speed increases that would generate more heat.
3Productivity
If multiple busses are implemented to increase parallelism, then throughput is improved, but device complexity increases
Solution Approach 1:
Each compute block is designed as a universal unit that can handle multiple data streams through its multiple accumulate busses. The same compute block structure is replicated across the array, with each block serving multiple functions by processing different data streams in parallel. This multi-functionality increases throughput without requiring fundamentally different block designs.
Solution Approach 2:
The patent changes the parameter of accumulate bus quantity from one to multiple within each column. This parameter change enables parallel accumulation operations, increasing throughput. The complexity increase is managed by maintaining regular, repetitive block structures that can be systematically instantiated.
4Loss of time
If multiple busses are implemented to enable sequence skipping, then latency is reduced, but manufacturing complexity increases
Solution Approach 1:
The calculation path is segmented into multiple parallel accumulation paths, each handled by a separate bus. This segmentation allows data to skip intermediate steps by accumulating directly through the appropriate bus, reducing the effective path length. The segmented structure is manufactured using standard repeating block patterns.
Solution Approach 2:
The patent uses replication of identical compute block templates, each containing the same set of accumulate busses. This copying approach simplifies manufacturing by allowing the same design to be repeatedly instantiated across the array, reducing fabrication complexity despite the increased internal parallelism.
Data Source
AI summary
Systems and methods are provided to enable parallelized multiply-accumulate operations in a systolic array. Each column of the systolic array can include multiple busses enabling independent transmission of input partial sums along the respective bus. Each processing element can include a plurality of interconnects to receive a plurality of inputs corresponding to the multiple busses. Each processing element of a given columnar bus can receive an input from a prior element of the given columnar bus at an active bus position and perform arithmetic operations on the input. Each processing element can further receive a plurality of inputs at passive bus positions and provide the plurality of inputs to subsequent processing elements without the plurality of inputs being processed by the processing element. Use of columnar busses can enable parallelization to increase speed or enable increased latency at individual processing elements.