Tensorized DMA Descriptors Reduce Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Direct Memory Access (DMA) transfers between accelerators and system memory in neural network training and inference are inefficient due to high overhead from retrieving multiple memory descriptors, especially for small data transfers, which limits data transfer size and increases latency.
Innovation Solution
Tensorization of memory descriptors is introduced, transforming lower-order data into higher-order data to reduce the number of descriptors needed, allowing for larger data transfers with reduced overhead and memory bandwidth usage, and enabling efficient data transfer with multidimensional strides and non-uniform striding patterns.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple memory descriptors are used for DMA transfers, then data transfer flexibility is improved, but DMA overhead increases
Solution Approach 1:
The patent merges multiple memory descriptors into a single tensorized template descriptor that can generate multiple child descriptors. This combines the functionality of multiple descriptors while reducing the overhead of retrieving and processing each descriptor individually, thereby maintaining data transfer flexibility while reducing DMA overhead.
Solution Approach 2:
The patent performs preliminary action by pre-configuring a tensorized template descriptor with all necessary parameters (stride information, data dimensions, memory addresses) before actual DMA transfers. This allows the template descriptor to generate multiple child descriptors on-demand, reducing the need to retrieve multiple separate descriptors during data transfers.
2Reliability
If multiple memory descriptors are retrieved for small data transfers, then data transfer completeness is improved, but memory bandwidth usage increases
Solution Approach 1:
The patent merges multiple descriptor retrievals into a single template descriptor retrieval. The template descriptor contains all necessary information to generate multiple child descriptors locally, eliminating the need to fetch multiple separate descriptors from memory, thereby reducing memory bandwidth usage while maintaining complete data transfer capability.
3Measurement precision
If multiple memory descriptors are used, then data transfer accuracy is improved, but device complexity increases
Solution Approach 1:
The patent merges the management of multiple descriptors into a single template descriptor with embedded stride and dimension information. This reduces descriptor management complexity by eliminating the need to track and manage multiple separate descriptors, while the template's structured parameters ensure accurate data transfer through systematic child descriptor generation.
4Productivity
If data transfer size is increased, then throughput is improved, but descriptor retrieval overhead increases
Solution Approach 1:
The patent performs preliminary action by pre-configuring the tensorized template descriptor with comprehensive transfer parameters including stride information, data dimensions, and memory addresses. This allows large data transfers to be executed by generating multiple child descriptors from the single template, avoiding repeated descriptor retrieval overhead and improving throughput.
Data Source
AI summary
Techniques to reduce overhead in a direct memory access (DMA) engine can include processing descriptors from a descriptor queue to obtain a striding configuration to generate tensorized memory descriptors. The striding configuration can include, for each striding dimension, a stride and a repetition number indicating a number of times to repeat striding in the corresponding striding dimension. One or more sets of tensorized memory descriptors can be generated based on the striding configuration. Data transfers are then performed based on the generated tensorized memory descriptors.