Neural Network Coefficient Compression via Matrix Sub-structuring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks require significant memory bandwidth to read large sets of coefficients, leading to high latency and computational demands, especially since many coefficients are zero and result in unnecessary element-wise multiplications.
Innovation Solution
A method to compress neural networks by rearranging matrices of coefficients into sub-matrices with higher non-zero density, allowing for efficient matrix multiplications using only non-zero coefficients, thereby reducing memory bandwidth and computational load.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If large sets of coefficients are stored and processed in neural networks, then computational accuracy is maintained, but memory bandwidth requirements and latency increase significantly
Solution Approach 1:
The patent extracts and removes zero coefficients from the coefficient sets before storage and processing. By identifying and eliminating these redundant zero values, the system reduces memory bandwidth requirements and processing latency while maintaining computational accuracy, as zero coefficients do not contribute to the final computation results
Solution Approach 2:
The patent changes the representation parameter of coefficients from fixed-precision floating-point format to variable-precision formats (such as integer or reduced-precision floating-point). This parameter change reduces the memory bandwidth required to transport coefficients and decreases the computational complexity of operations while maintaining sufficient accuracy for the neural network's performance
2Reliability
If all coefficients including zeros are processed in matrix multiplications, then complete computational coverage is achieved, but computational demands and energy consumption increase
Solution Approach 1:
The patent extracts zero coefficients from the computation process entirely. By identifying positions of zero coefficients in advance and skipping computations at these positions, the system eliminates unnecessary multiply-accumulate operations, thereby reducing energy consumption while maintaining computational completeness for all non-zero coefficients
Solution Approach 2:
The patent applies partial action by performing computations only on non-zero coefficients rather than all coefficients. This selective computation approach processes exactly the necessary portion (non-zero coefficients) and skips the excessive portion (zero coefficients), optimizing energy efficiency without compromising the neural network's computational accuracy
3Device complexity
If coefficient data is stored in standard matrix format, then data structure simplicity is maintained, but memory bandwidth requirements are high due to inclusion of zero values
Solution Approach 1:
The patent segments the coefficient matrix into multiple sub-matrices or blocks, where each block contains primarily non-zero coefficients. This segmentation allows for more efficient storage and access patterns, reducing the amount of data that needs to be loaded into memory at any given time and thereby reducing memory access time while maintaining reasonable data structure complexity
Solution Approach 2:
The patent changes the data format parameter from standard floating-point representation to more compact representations (such as integer formats or specialized compressed formats). This parameter change reduces the memory bandwidth required to transfer coefficient data while maintaining the necessary precision for neural network computations
Data Source
AI summary
A neural network is compressed by selecting two or more adjacent layers, each having one or more input channels and one or more output channels, a first layer performing a first operation and a second layer performing a second operation. First and second matrices representative of sets of coefficients of the first and second layers are determined, having a plurality of elements representative of non-zero values and a plurality of elements representative of zero values. An array is formed comprising the first matrix and the second matrix by aligning the columns or rows of the first matrix that are representative of the output channels of the first layer with the columns or rows of the second matrix that are representative of the input channels of the second layer. The rows and/or columns of the array are rearranged into respective first and second sub-matrices. A compressed neural network is then outputted.