DNN Weight Matrix Compression Using Low Displacement Rank
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep Neural Networks (DNNs) face high computational complexity, making them unsuitable for resource-limited devices due to their massive number of parameters, which existing compression methods like sparsity-based and low-rank approximation fail to effectively address.
Innovation Solution
The proposed method uses Low Displacement Rank (LDR) approximation to decompose layer weight matrices of pre-trained DNNs into structured matrices, enabling simultaneous compression and reduced inference complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep neural networks with massive parameters are used to achieve high accuracy, then model performance is improved, but inference complexity and computational requirements increase
Solution Approach 1:
The patent segments the weight matrices into multiple low-displacement-rank components, representing each matrix as a sum of structured matrices with displacement ranks significantly smaller than the original dimensions. This segmentation reduces the number of parameters while maintaining approximation accuracy, resolving the contradiction between model performance and inference complexity.
Solution Approach 2:
The patent changes the parameter representation from full-rank weight matrices to low-displacement-rank structured matrices defined by displacement operators and core coefficients. This parameter transformation reduces the storage requirements and computational complexity from O(m×n) to O((m+n)×k) where k is the displacement rank, enabling deployment on resource-limited devices while preserving model accuracy.
2Measurement precision
If full precision weight matrices are stored and processed, then model accuracy is maintained, but memory usage and storage requirements increase
Solution Approach 1:
The weight matrices are segmented into structured components with low displacement ranks, allowing the same information to be represented with fewer parameters. This segmentation enables efficient storage while maintaining the ability to reconstruct accurate weight matrices during inference, reducing memory usage without sacrificing model accuracy.
Solution Approach 2:
The patent transforms the storage format from storing all weight matrix elements to storing only the displacement operators and core coefficients that define the low-displacement-rank structure. This parameter change dramatically reduces the quantity of data that needs to be stored and transmitted, while the full precision can be recovered during inference through the structured reconstruction process.
3Device complexity
If low displacement rank approximation is applied to compress weight matrices, then model size is reduced, but potential loss of information and accuracy may occur
Solution Approach 1:
The patent applies low-displacement-rank approximation as a partial action that captures the most significant information in the weight matrices while discarding less important details. By optimizing the displacement rank and using iterative refinement processes, the method achieves sufficient accuracy for practical applications without requiring full precision, thus reducing model size while minimizing information loss.
Solution Approach 2:
The patent employs feedback mechanisms during the approximation process, where the quality of the low-displacement-rank representation is evaluated and used to adjust the displacement rank and optimization parameters. This feedback loop ensures that the compressed representation maintains adequate accuracy by iteratively improving the approximation until the information loss is within acceptable thresholds.
Data Source
AI summary
A method and an apparatus for performing deep neural network compression use an approximation training set along with information, such as in matrices representing weights, biases and non-linearities, to iteratively compress a pre-trained deep neural network by low displacement rank based approximation of the network layer weight matrices. The low displacement rank approximation allows for replacement of an original layer weight matrices of the pre-trained deep neural network as the sum of a small number of structured matrices, allowing compression and low inference complexity.