Interleaved Tensor Transfer Across Parallel Memory Channels
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning operations involving large amounts of tensor data and frequent exchanges between local and external memory strain the bandwidth of memory channels, leading to inefficiencies in data transfer.
Innovation Solution
A hardware accelerator system that interleaves tensor data across multiple external memory modules, utilizing concurrent data transactions through independent memory channels to distribute bandwidth and enhance transfer speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If tensor data is transferred through a single memory channel, then the system is simpler to implement, but the data transfer speed is limited and bandwidth is insufficient
Solution Approach 1:
The patent divides the tensor data into multiple sub-tensors and distributes them across multiple independent memory channels. Each channel handles a portion of the data concurrently, thereby increasing overall transfer speed while maintaining manageable complexity through modular channel design
Solution Approach 2:
The patent transitions from single-channel sequential transfer to multi-channel parallel transfer by adding the dimension of concurrency. Multiple memory channels operate simultaneously to transfer different sub-tensors, effectively increasing data transfer throughput without proportionally increasing system complexity
2Productivity
If large amounts of tensor data are exchanged frequently between local and external memory, then machine learning operations can be performed, but the memory channel bandwidth is strained
Solution Approach 1:
Large tensor data is segmented into smaller sub-tensors that can be distributed across multiple memory channels. This segmentation allows frequent data exchanges to be handled in parallel, reducing the burden on any single channel while maintaining high overall throughput for machine learning operations
Solution Approach 2:
Multiple memory channels are merged into a unified transfer system that handles large tensor data exchanges collectively. The combined bandwidth of multiple channels addresses the high data volume requirements of frequent machine learning operations without overloading individual channels
Data Source
Figure 1
Figure 2A~2B
Figure 3
AI summary
A device includes a direct memory access (DMA) controller comprising DMA channels, a bridge circuit configured to couple the DMA channels to memory channels coupled to respective memory modules, and a local memory unit. The DMA controller is configured to transfer tensor data between the local memory unit and the memory modules via the DMA channels and the memory channels using concurrent data transactions, the tensor data is stored and addressed as parts of a single tensor in the local memory unit, and the tensor data is interleaved onto the memory modules and is stored and addressed as sub-tensors in respective memory modules.