Serial DMA Weight Transfer for Neural Accelerator Bandwidth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In neural networks that process large amounts of data, the memory interface is often bottlenecked when loading weight values to multiple accelerators in parallel, leading to reduced throughput due to the need for each accelerator to access external memory sequentially.
Innovation Solution
Implementing a serial DMA transfer mechanism where weight values are loaded into one accelerator and then cascaded to others, with event-synchronization ensuring data availability and freeing up memory space, while using striped input batching to distribute input data efficiently across accelerators.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If each neural network accelerator loads weight values from external memory independently, then each accelerator can access the weight values simultaneously, but the memory interface is occupied for an extended period and bandwidth is reduced
Solution Approach 1:
The patent creates copies of weight values across multiple accelerators through a cascading transfer mechanism. The first accelerator receives weight values from external memory, then sequentially transfers copies to subsequent accelerators. This allows all accelerators to have local copies for simultaneous access, eliminating the need for repeated external memory accesses while preserving bandwidth.
Solution Approach 2:
The patent performs preliminary action by loading weight values into the first accelerator before the computation begins. The serial DMA transfer mechanism pre-loads and cascades weight values to all accelerators in advance, so that when computation starts, all accelerators already have the weight values available in their local memory, avoiding delays during computation.
2Productivity
If weight values are transferred serially from one accelerator to the next, then external memory bandwidth is preserved, but the transfer time may increase
Solution Approach 1:
The patent implements continuous useful action through the cascading transfer mechanism. While the first accelerator is computing with its weight values, the second accelerator is simultaneously receiving weight values from the first, and the third is receiving from the second, and so on. This continuous pipeline of data transfer ensures that accelerators are always productive—either computing or receiving data—minimizing idle time despite the serial transfer nature.
Solution Approach 2:
The patent uses dynamic timing control where each accelerator is configured to receive weight values at optimized intervals. The system dynamically adjusts the transfer timing so that weight values are cascaded to accelerators just in time for computation, balancing the serial transfer constraint with minimal waiting time. Event-synchronization signals coordinate this dynamic timing to prevent bottlenecks.
3Reliability
If event-synchronization is used to ensure weight values are available, then data availability is guaranteed, but memory space management becomes more complex
Solution Approach 1:
The patent implements feedback through event-synchronization signals that flow in the opposite direction of the weight value cascade. Each accelerator generates an event signal indicating when it has finished processing or is ready to receive weight values. These feedback signals allow the system to track the state of each accelerator and coordinate transfers efficiently, ensuring data availability while managing memory space through event-driven control.
Solution Approach 2:
The patent enables self-service where each accelerator autonomously manages its own weight value reception and computation. Accelerators monitor their own readiness state and generate event signals when they need weight values or have completed processing. This self-service approach simplifies the overall synchronization mechanism compared to centralized control, as each unit manages its own data availability needs independently.
Data Source
AI summary
An acceleration engine with multiple accelerators may share a common set of data that is used by each accelerator to perform computations on input data. The set of shared data can be loaded into the acceleration engine from an external memory. Instead of accessing the external memory multiple times to load the set of shared data into each accelerator, the external memory can be accessed once using direct memory access to load the set of shared data into the first accelerator. The set of shared data can then be serially loaded from one accelerator to the next accelerator in the acceleration engine using direct memory access. To achieve data parallelism and reduce computation time, a runtime driver may split the input data into data batches, and each accelerator can perform computations on a different batch of input data with the common set of shared data.