Grid DMA Controller for Tensor Data Distribution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning accelerators face performance bottlenecks due to high computations and data movement in convolutional neural networks, particularly in edge devices with strict power consumption constraints, such as AR/VR headsets and mobile devices, where they struggle to perform well for real-time inference exercises.
Innovation Solution
A grid direct-memory access (DMA) controller is used to efficiently read and write tensor data in bursts, with source and destination memory controllers executing loop instructions in lockstep to generate addresses, allowing for arbitrary alignment and optimizing data transfer between tensor processor units within the ML accelerator.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing ML accelerators use high compute parallelism and optimized data orchestration, then processing speed of convolutional layers is improved, but power consumption increases making them unsuitable for edge devices
Solution Approach 1:
The patent introduces a DMA controller as an intermediary component that manages data transfer between memory and tensor processor units independently of the main processing units. This mediator handles data orchestration tasks, allowing the tensor processor units to focus on computation while reducing overall system power consumption through specialized, optimized data movement pathways.
Solution Approach 2:
The system segments data transfer operations into discrete bursts handled by the DMA controller, separating data movement functions from computation functions. This segmentation allows independent optimization of each function, enabling efficient data transfer without requiring continuous high-power computation resources.
2Productivity
If tensor data is transferred in bursts with arbitrary alignment between source and destination memory, then data transfer efficiency is improved, but controller complexity increases
Solution Approach 1:
The DMA controller executes loop instructions in lockstep to automatically generate source and destination addresses without external intervention. This self-service mechanism handles arbitrary alignment internally through iterative address generation, eliminating the need for complex external address management while maintaining efficient burst transfer capabilities.
Solution Approach 2:
The system performs preliminary address generation and alignment calculations before actual data transfer begins. By pre-computing address sequences using loop instructions, the DMA controller prepares all necessary addressing information in advance, simplifying the actual transfer operation and reducing runtime complexity.
3Adaptability or versatility
If source and destination memory addresses are arbitrarily aligned, then flexibility in data placement is improved, but address generation complexity increases
Solution Approach 1:
The address generation system uses dynamic loop instructions that can adapt to any alignment requirement between source and destination memory. The iterative nature of the loop allows the system to handle arbitrary address alignments flexibly, adjusting address sequences on-the-fly without requiring complex static addressing logic.
Data Source
AI summary
In one embodiment, a method for tensor data distribution using a direct-memory access agent includes generating, by a first controller, source addresses indicating locations in a source memory where portions of a source tensor are stored. A second controller may generate destination addresses indicating locations in a destination memory where portions of a destination tensor are to be stored. The direct-memory access agent receives a source address generated by the first controller and a destination address generated by the second controller and determines a burst size. The direct-memory access agent may issue a read request comprising the source address and the burst size to read tensor data from the source memory and may store the tensor data into an alignment buffer. The direct-memory access agent then issues a write request comprising the destination address and the burst size to write data from the alignment buffer into the destination memory.