Neural Network Systolic Array for Low-Latency On-Chip Rearrangement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The latency caused by data rearrangement between layers in systolic-array-based artificial neural network accelerators degrades processing performance due to the need to transfer tensors between internal and external memory.
Innovation Solution
An artificial neural network processing accelerator with a systolic array structure that includes internal memory banks and rearrangement interfaces, utilizing multiplexers and a data flow control unit to manage data flow and address generation, allowing intra-chip data rearrangement without external memory access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data rearrangement is performed by transferring tensors between internal memory and external memory, then data can be rearranged to fit subsequent layer inputs, but access to external memory causes high latency that degrades processing performance
Solution Approach 1:
The patent introduces a new spatial dimension by adding intermediate buffer memory between the systolic array and external memory. This buffer memory provides an additional storage layer that enables data rearrangement operations to be performed on-chip without requiring transfers to external memory, thus resolving the latency issue while maintaining adaptability.
Solution Approach 2:
The patent introduces buffer memory as an intermediary component between the systolic array and external memory. This intermediary structure facilitates data rearrangement operations by providing temporary storage space on-chip, allowing the system to perform adaptive data formatting without the performance penalty of external memory access.
2Ease of operation
If data is stored layer by layer in internal memory, then memory access is simplified, but the size mismatch between input and output tensors of different layers requires additional rearrangement operations
Solution Approach 1:
The patent segments the memory system into multiple functional regions: input buffer memory, output buffer memory, and intermediate buffer memory. This segmentation allows different types of data operations to occur in dedicated regions, simplifying memory access patterns while enabling complex data rearrangement through coordinated access to different segments.
Solution Approach 2:
The patent adds a temporal dimension to memory organization by introducing multi-cycle buffer stages. Data can be staged through multiple buffer regions across different clock cycles, allowing simple sequential access patterns to evolve into complex rearrangement operations without increasing the complexity of individual access commands.
Data Source
AI summary
Disclosed herein is an artificial neural network processing accelerator based on a systolic array. The artificial neural network processing accelerator based on a systolic array includes a processing element array, internal memory for storing input/output data of the processing element array, and a data flow control unit for performing control to deliver an input tensor and weight data from the internal memory to the processing element array in each operation cycle and to store an output tensor from the processing element array in the internal memory. The internal memory may include N memory banks respectively corresponding to N rows of the processing element array and further include a rearrangement interface for connecting input/output between the N rows of the processing element array and the N memory banks.