Sparse Weight Matrix Layout for Lower Inference Memory Usage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional inference devices fail to effectively reduce memory usage due to biased distribution of non-zero elements in weight matrices, leading to inefficient memory allocation and increased circuit scale, especially in digital circuit implementations.
Innovation Solution
The proposed inference device stores only non-zero weights and their position information in memory, ensuring uniform distribution of non-zero elements across rows and columns, thereby optimizing memory usage and reducing the circuit scale.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional inference devices store weight matrices with biased non-zero element distribution, then memory allocation becomes inefficient and circuit scale increases, but simply reducing non-zero elements does not necessarily reduce memory usage
Solution Approach 1:
The patent transforms the weight matrix by applying permutation operations to rows and columns, changing the positional parameters of non-zero elements while maintaining the matrix's mathematical properties. This reparameterization achieves uniform distribution of non-zero elements, enabling efficient memory storage and reduced circuit scale without altering the underlying weight values or network performance
Solution Approach 2:
The patent segments the weight matrix into non-zero elements and zero elements, storing only the non-zero elements in memory along with their position information. This segmentation approach reduces memory usage by eliminating storage of redundant zero values while the uniform distribution ensures efficient memory allocation patterns
2Quantity of substance
If the number of non-zero elements in weight matrices is reduced, then the amount of operation and memory decrease, but the distribution of non-zero elements becomes biased leading to inefficient memory allocation
Solution Approach 1:
The patent applies permutation transformations to change the positional parameters of non-zero elements in the weight matrix. By systematically repositioning non-zero elements through row and column permutations, the patent achieves uniform distribution that optimizes memory allocation efficiency while maintaining the reduced count of non-zero elements
3Device complexity
If weight matrices are stored with uniform distribution of non-zero elements, then memory usage is optimized and circuit scale is reduced, but additional processing is required to achieve uniform distribution
Solution Approach 1:
The patent performs preliminary permutation operations on the weight matrix during the model preparation or training phase to achieve uniform distribution of non-zero elements before deployment. This preliminary action ensures that the inference device operates with optimized memory allocation and reduced circuit scale without incurring processing overhead during actual inference operations
Data Source
AI summary
Included in an inference device of a neural network are: a memory for storing a layer on an input side and weight data for generating a matrix multiplication using the layer on the input side; and a processor for generating a layer on an output side, by using the layer on the input side and the weight data for generating the matrix multiplication using the layer on the input side, in which, out of a plurality of rows and columns including zero elements and non-zero elements in the weight data, the memory stores only non-zero weights and position information of the non-zero weights and the number of non-zero elements in each of the rows is approximately the same. Therefore, it is possible to reduce the amount of memory in a memory for storing learned weight data of a weight matrix. It is also possible to reduce the amount of memory of a memory for storing positions of non-zero weight elements of the weight matrix.