Systolic Array Weight Sparsity Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques fail to effectively exploit fine-grained structured weight sparsity in systolic arrays-based computing systems and native convolutions, leading to inefficiencies in execution time and energy usage in deep neural networks.
Innovation Solution
The method involves storing indices of non-zero weights in an index register file and the weights themselves in a register file within each processor element of a systolic array, allowing for selective processing of input values corresponding to non-zero weights through multiply-accumulate operations, optimizing data storage and processing by reusing input values for multiple operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing techniques are used in systolic arrays, then the system can perform basic deep neural network operations, but execution time and energy consumption are inefficient due to inability to exploit fine-grained structured weight sparsity
Solution Approach 1:
The patent segments the weight matrix into fine-grained blocks (e.g., 4x4 submatrices) and processes only the non-zero elements within each block. This segmentation allows the systolic array to skip zero-weight multiplications and focus computation only on relevant portions, reducing both execution time and energy consumption while maintaining computational accuracy.
Solution Approach 2:
The patent implements local quality by storing and processing only the non-zero weight values and their positions within each fine-grained block. The system maintains different handling for zero and non-zero elements, selectively activating only those processor elements that need to perform MAC operations on non-zero weights, thereby optimizing resource utilization and reducing energy waste.
2Productivity
If all input values are processed through all processor elements, then complete matrix multiplication is achieved, but resource utilization is inefficient when weights are sparse
Solution Approach 1:
The patent introduces dynamic activation of processor elements based on the sparsity pattern of weights. Instead of all processor elements being actively involved in every computation, the system dynamically activates only those elements corresponding to non-zero weight positions. This dynamic adaptation allows the system to maintain high computational throughput while improving resource utilization efficiency.
Solution Approach 2:
The patent extracts and stores only the non-zero weight values and their indices in separate data structures (weight matrices and index matrices). This extraction allows the system to ignore zero-weight entries entirely during computation, reducing the effective computational workload and improving both throughput and resource utilization by processing only essential operations.
3Device complexity
If dense storage and processing of weights is used, then data structure simplicity is maintained, but memory bandwidth and storage capacity are wasted on zero values
Solution Approach 1:
The patent extracts zero values from the weight matrix and stores only non-zero weight values along with their positions in compressed data structures. This extraction reduces the quantity of data that needs to be stored and transmitted, significantly lowering memory bandwidth consumption and storage requirements while maintaining the ability to perform accurate matrix multiplication through the index information.
Solution Approach 2:
The patent changes the storage parameter from dense format to sparse format, where weights are stored only at non-zero positions with explicit index information. This parameter change allows the system to reduce memory bandwidth consumption and storage capacity requirements while maintaining computational functionality. The index register file stores position information that enables reconstruction of the full weight matrix when needed.
Data Source
AI summary
Indices of non-zero weights may be stored in an index register file included within each of a plurality of processor elements in a systolic array. Non-zero weights may be stored in a register file associated with the index register file. Input values (e.g., dense input values) corresponding to a single block in a data structure may be sent to the plurality of processor elements. Those of the input values corresponding to the indices of non-zero weights in the index register file may be selected for performing multiply-accumulate (“MAC”) operation based on sending the plurality of input values to one or more of the plurality of processor elements. The indices of the plurality of non-zero weight are stored in an index data stick. The values of the plurality of non-zero weights are stored in a value data stick.