Neural Network Weight Approximation via Codebook Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Convolutional neural networks (CNNs) face challenges in improving accuracy when using low-rank approximation methods for weight parameters, especially in higher layers, due to sparse or inconsecutive weights, which limits the effectiveness of existing compression techniques.
Innovation Solution
A method involving a division unit that divides weight parameters into groups and an encoding unit that approximates and encodes these parameters using a codebook, allowing for high-accuracy approximation of sparse weights by reconstructing them with a codebook coefficient.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If low-rank approximation methods are used for weight parameters in higher layers of CNN, then compression ratio is improved, but approximation accuracy deteriorates due to sparse or inconsecutive weights
Solution Approach 1:
The weight parameters are divided into multiple groups based on their magnitude or position, allowing different approximation strategies to be applied to different segments. This segmentation enables the sparse weight regions to be handled differently from dense regions, improving overall approximation accuracy while maintaining compression.
Solution Approach 2:
Different approximation methods or parameters are applied to different regions of the weight matrix based on local characteristics. Sparse regions use one approximation strategy while dense regions use another, optimizing the balance between compression and accuracy for each local region rather than applying a uniform approach.
2Volume of stationary object
If direct product approximation is used for weight parameters, then memory requirement is reduced, but accuracy is compromised especially in higher layers with sparse weights
Solution Approach 1:
The weight parameters are segmented into multiple groups, and each group is approximated independently using codebooks. This allows the model to maintain lower memory requirements through compression while preserving important weight information by treating different segments with appropriate approximation levels.
Solution Approach 2:
Codebooks are created as compressed representations (copies) of the original weight parameters. These codebooks store the essential information in a reduced form, allowing the model to operate with lower memory requirements while still reconstructing the weight parameters with acceptable accuracy when needed.
Data Source
AI summary
According to the present disclosure, a weight parameter of a neural network is divided into a plurality of portions having a certain size and approximation is individually performed on the portions using a weighted sum of the codebook vectors.