Neural Network Accelerator Weight Merging and Instruction Packaging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network processing systems face overhead and latency issues due to frequent direct memory access (DMA) operations between the host computer system and the neural network accelerator, particularly when transferring data and weights for multiple layers of a neural network.
Innovation Solution
The method involves pre-assembling all input data and parameters for a neural network in shared memory before processing, merging weight matrices into a single contiguous block, and using a package of per-layer instructions to reduce DMA operations and optimize data access, allowing the neural network accelerator to process data efficiently without frequent memory reloads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data and weights are transferred frequently between host computer system and neural network accelerator via DMA operations, then the neural network accelerator can access updated data and weights, but the overhead and latency increase significantly
Solution Approach 1:
The host computer system pre-assembles all input data and parameters for the neural network in shared memory before processing begins. Weight matrices are merged into a single contiguous block, and all data is prepared in advance to eliminate the need for frequent DMA operations during neural network execution, thereby reducing overhead and latency while ensuring data access accuracy
2Ease of operation
If weight matrices are stored separately for each layer, then each layer can access its specific weights, but the number of DMA operations and memory access overhead increases
Solution Approach 1:
Weight matrices from multiple neural network layers are merged into a single contiguous block in shared memory. Each layer's weights are organized sequentially within this unified structure, allowing the neural network accelerator to access all weights through continuous memory access without requiring separate DMA operations for each layer, thus reducing memory access complexity while maintaining ease of layer-specific weight retrieval
3Productivity
If the neural network accelerator processes each layer with separate data loading, then each layer can be processed independently, but the processing speed decreases due to frequent memory reloads
Solution Approach 1:
The neural network accelerator processes multiple layers continuously without interruption by keeping all weight matrices and input data readily available in shared memory. The merged weight structure enables seamless transitions between layers without memory reloads, maintaining continuous processing flow and eliminating idle time between layers, thereby improving both processing throughput and speed
Data Source
Figure 1
Figure 2
Figure 3
AI summary
In the disclosed methods and systems for processing in a neural network system, a host computer system (402) writes (602) a plurality of weight matrices associated with a plurality of layers of a neural network to a memory (226) shared with a neural network accelerator (238). The host computer system further assembles (610) a plurality of per-layer instructions into an instruction package. Each per-layer instruction specifies processing of a respective layer of the plurality of layers of the neural network, and respective offsets of weight matrices in a shared memory. The host computer system writes (612, 614) input data and the instruction package to the shared memory. The neural network accelerator reads (702) the instruction package from the shared memory and processes (702-712) the plurality of per-layer instructions of the instruction package.