Sparse Neural Matrix-Vector Multiplication with Zero-Skipping Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Matrix by vector multiplication in artificial neural networks is computationally intensive due to large, sparse weight matrices, leading to inefficient use of computational resources and increased storage costs from handling zero entries.
Innovation Solution
A method that compresses weight values and indices into a compressed stream by removing zero entries and shifting non-zero entries in the weight matrix, allowing for parallel processing across multiple neurons with multiple inputs, thereby maximizing MAC hardware utilization and reducing storage needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If weight matrices are stored and processed in conventional format, then complete data representation is achieved, but storage space increases and processing efficiency decreases due to sparse zero entries
Solution Approach 1:
The patent extracts and removes zero entries from the weight matrix, storing only non-zero values along with their corresponding row and column indices. This extraction principle directly reduces storage requirements while maintaining the functional integrity of the matrix for M×V operations.
Solution Approach 2:
The patent segments the weight matrix into blocks and processes them individually, allowing efficient handling of sparse data. By dividing the matrix into manageable segments and processing only non-zero elements within each segment, the system improves processing efficiency and reduces computational overhead.
2Productivity
If conventional M×V procedure is used, then standard computation is performed, but computational resources are consumed and processing time increases
Solution Approach 1:
The patent extracts only non-zero elements from the weight matrix for processing, eliminating unnecessary computations involving zero entries. This extraction approach directly reduces computational resource consumption and accelerates M×V operations by focusing only on meaningful data.
Solution Approach 2:
The patent performs preliminary compression of the weight matrix into a compact format with non-zero elements and their indices before the actual M×V operation. This preliminary action prepares the data in an optimized state, enabling faster processing and reducing computational resource requirements during the main computation phase.
3Reliability
If zero entries are stored in weight matrix, then complete matrix structure is maintained, but storage cost increases and operation efficiency decreases
Solution Approach 1:
The patent removes zero entries from storage while maintaining data integrity through careful tracking of non-zero elements and their positions using index arrays. This extraction approach preserves all necessary information for accurate M×V operations while eliminating wasteful storage of redundant zero values.
Solution Approach 2:
The patent creates index copies (row indices and column indices) that reference the positions of non-zero elements in the original matrix structure. These index copies enable reconstruction and processing of the matrix data without physically storing all zero entries, thus maintaining data integrity with reduced storage requirements.
Data Source
AI summary
A method for matrix by vector multiplication, applied in an artificial neural network system, is disclosed. The method comprises: compressing a plurality of weight values in a weight matrix and indices of an input vector into a compressed main stream; storing M sets of synapse values in M memory devices; and, performing reading and MAC operations according to the M sets of synapse values and the compressed main stream to obtain a number M of output vectors. The step of compressing comprises: dividing the weight matrix into a plurality of N×L blocks; converting entries of a target block and corresponding indices of the input vector into a working block and an index matrix; removing zero entries in the working block; shifting non-zero entries row-by-row to one of their left and right sides in the working block; and, respectively shifting corresponding entries in the index matrix.