DMA Multicast Data Replication for Neural Network Accelerators
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network accelerators face inefficiencies due to the need for complex shifting logic and multiple memory reads when processing large datasets, which can lead to system memory throughput limitations and increased latency.
Innovation Solution
Implementing a direct memory access (DMA) engine with multicast capabilities to reduce memory reads and writes, allowing for efficient data transfer by writing unused padding data to a null device address, thereby maintaining data consistency and minimizing system memory throughput impact.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is replicated across multiple rows of the systolic array, then processing capability is improved, but memory read operations increase
Solution Approach 1:
The patent combines multiple read operations into a single read operation by replicating data in a buffer before distribution. The DMA engine reads data once and replicates it to multiple destinations (rows of the systolic array), merging what would otherwise be multiple separate read operations into one, thereby reducing memory access time while maintaining the ability to supply data to multiple processing rows simultaneously
2Manufacturing precision
If complex shifting logic is implemented to rearrange data, then data alignment is improved, but device complexity increases
Solution Approach 1:
The patent performs data alignment and rearrangement in advance during the data loading phase, before the actual neural network computation begins. The DMA engine pre-positions data in the buffer with proper alignment for each row, so that during computation, data can be streamed directly without requiring complex real-time shifting logic. This preliminary action eliminates the need for complex shifting circuits during the critical computation phase
3Loss of information
If multiple reads are performed from system memory, then data availability is improved, but system memory throughput decreases
Solution Approach 1:
The patent creates copies of the data in a local buffer memory before distribution to multiple processing rows. Instead of reading from system memory multiple times, the DMA engine performs a single read to the buffer and then replicates the data locally. This copying approach ensures data availability for all rows while minimizing system memory access to a single read operation, thereby preserving system memory throughput
Data Source
AI summary
A direct memory access (DMA) engine can be used to multicast data from system memory to a target memory for loading into an array. The DMA engine may include a controller that is configured to receive a data transfer request, and generate a set of write operations for the output interface. The set of write operations can include, for each of multiple partitions of the target memory, a write operation to write usable data from the multicast data to an address offset in the corresponding partition, and an additional write operation to write filler data from the multicast data to a null device address.