Tensorized Direct Memory Access Descriptors for Neural Network Data Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing direct memory access (DMA) techniques for neural network training and inference incur significant overhead due to the need to retrieve multiple memory descriptors, especially for smaller data transfers between accelerators and system memory, which limits the efficiency of data transfer and increases latency.
Innovation Solution
The proposed solution is to tensorize memory descriptors, transforming lower-order data into higher-order data, allowing a single tensorized descriptor to facilitate hundreds or thousands of DMA transfers, thereby reducing overhead and memory bandwidth requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional memory descriptors are used for DMA transfers, then each data transfer can be completed, but the overhead increases significantly for smaller data transfers
Solution Approach 1:
The patent combines multiple individual memory descriptors into a single tensorized memory descriptor that can represent and manage multiple data transfers simultaneously. This merging reduces the number of descriptor retrieval operations from the descriptor buffer, thereby reducing overhead and latency for multiple small data transfers while maintaining the ability to complete each transfer efficiently.
2Productivity
If multiple memory descriptors are retrieved for each data transfer, then data transfer can be performed, but memory bandwidth is consumed for descriptor retrieval
Solution Approach 1:
The patent merges multiple descriptor retrieval operations into a single descriptor structure that can be processed once, eliminating redundant memory bandwidth consumption associated with retrieving the same descriptor information multiple times for similar data transfers.
Solution Approach 2:
The tensorized memory descriptor is designed to serve multiple functions simultaneously - it can represent multiple data transfers, specify different source and destination addresses, and define various transfer parameters all within a single descriptor structure, reducing the need for multiple specialized descriptor retrievals.
3Ease of operation
If individual memory descriptors are used for each transfer, then precise control over each transfer is achieved, but the complexity of descriptor management increases
Solution Approach 1:
The patent merges the functionality of multiple descriptors into a single tensorized descriptor structure, reducing the number of descriptors that need to be stored and managed in the descriptor buffer. This simplifies descriptor management while maintaining precise control over each individual data transfer through the integrated descriptor fields.
Data Source
AI summary
To reduce direct memory access (DMA) overhead, a tensorized descriptor can be used to generate a series of memory descriptors to perform a series of DMA data transfers. The tensorized descriptor may include attributes such as a stride and a memory descriptor template, which can be used to generate the series of memory descriptors. Hence, instead of having to retrieve each of the memory descriptors to perform the series of DMA transfers, a single tensorized descriptor can be retrieved to perform a series of data transfers.