Weight Matrix Formatting for Neural Network Accelerators
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network weight matrix formatting methods lead to increased memory access latency and performance degradation due to random access patterns when multiplying sparse weight matrices with input vectors, especially in parallel processing environments where multiple processing elements access memory independently.
Innovation Solution
The method involves formatting the weight matrix into sub-matrices, where weight information from respective columns is combined and organized to enable sequential access of input vector elements, allowing for efficient distribution of calculations across processing elements and minimizing memory access overhead by using a global buffer and local buffers to store and manage input vector elements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional weight matrix formatting methods (CSR, CSC, CISR) are used, then the weight matrix can be stored in memory, but memory access latency increases due to random access patterns when multiplying with input vectors
Solution Approach 1:
The weight matrix is divided into multiple sub-matrices, where each sub-matrix corresponds to a specific row range. This segmentation allows processing elements to work on specific sub-matrices independently, improving memory access efficiency by reducing the scope of random accesses needed for each processing element.
Solution Approach 2:
The weight matrix is pre-formatted into a specialized structure with metadata including row pointers, column pointers, and value arrays before being loaded into memory. This preliminary formatting organizes the data so that during multiplication operations, processing elements can access required weights in a more sequential and predictable pattern, reducing memory access latency.
2Productivity
If multiple processing elements are used for parallel processing, then computation speed should increase, but memory access time increases because each PE needs to independently access input data
Solution Approach 1:
The input vector is divided into multiple segments, with each processing element responsible for a specific segment. The weight matrix is also segmented into corresponding sub-matrices. This segmentation allows each PE to process its assigned segment independently with localized memory accesses, avoiding the need for all PEs to compete for the same memory resources.
Solution Approach 2:
A control circuit acts as an intermediary between the processing elements and memory system. It manages the distribution of input data segments to appropriate PEs and coordinates memory access patterns, allowing multiple PEs to operate in parallel with minimized memory access conflicts and latency.
3Quantity of substance
If the weight matrix is sparse with many zero values, then storage size can be reduced by excluding zero values, but elements of the input vector become more discontinuously distributed, increasing memory latency
Solution Approach 1:
Zero values are extracted and removed from the weight matrix storage, keeping only non-zero values along with their position metadata. This extraction reduces storage size while the metadata structure is designed to maintain efficient access patterns by grouping non-zero weights that correspond to contiguous or near-contiguous input vector elements.
Solution Approach 2:
The storage format parameters are changed from storing all weights including zeros to storing only non-zero weights with metadata. The metadata includes row indices, column indices, and value arrays that are organized to maintain sequential access patterns during multiplication, thereby reducing memory latency despite the sparsity of the matrix.
Data Source
AI summary
A method for formatting a weight matrix including a plurality of sub matrices each being multiplied with an input vector may include sequentially adding weight information included in respective first columns of the plurality of sub matrices to formatted data; and sequentially adding weight information included in respective second columns of the plurality of sub matrices to the formatted data after the weight information from the first columns of the plurality of sub matrices. The weight information may be non-zero weight information.