Convolutional Neural Network Weight Compression via Low Displacement Rank
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep Neural Networks (DNNs) face high inference complexity due to their massive number of parameters, making them challenging for use in resource-limited devices, and existing compression methods like sparsity-based and low-rank approximation do not effectively reduce inference complexity or control sparsity patterns.
Innovation Solution
The method involves reshaping convolutional layer tensors into lower-dimensional matrices, approximating them using Low Displacement Rank (LDR) structures, and encoding these approximations for compression, allowing for a sum of structured matrices that reduce inference complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If DNNs use a huge number of parameters to achieve state of the art performance, then accuracy is improved, but inference complexity and computational cost increase
Solution Approach 1:
The patent segments the weight tensor into multiple sub-tensors and applies different compression techniques to each segment. This allows the network to maintain high accuracy in critical segments while reducing overall complexity through selective compression of less important weight components.
Solution Approach 2:
The patent transforms the weight parameters from their original high-dimensional form into a compressed representation using techniques like singular value decomposition and low-rank approximation. This parameter transformation reduces the number of parameters while preserving the essential information needed for accurate inference.
2Measurement precision
If DNNs have massive number of parameters for high performance, then model accuracy is improved, but transmission and storage requirements increase
Solution Approach 1:
The patent creates a compressed copy of the original weight tensor that can be transmitted and stored efficiently. This compressed representation captures the essential characteristics of the full model while requiring significantly less storage space and bandwidth for transmission.
Solution Approach 2:
The patent reduces the dimensionality of the parameter space by transforming weights from their original multi-dimensional tensor form into a lower-dimensional representation. This dimensional reduction maintains the functional equivalence of the network while dramatically reducing the quantity of parameters that need to be stored and transmitted.
3Quantity of substance
If existing compression methods use sparsity-based or low-rank approximation, then parameter quantity is reduced, but inference complexity is not effectively reduced and sparsity patterns cannot be controlled
Solution Approach 1:
The patent applies different compression strategies to different parts of the network based on their importance and characteristics. Critical layers maintain higher precision while less important layers undergo more aggressive compression, allowing the system to control sparsity patterns and manage inference complexity effectively across different regions of the network.
Data Source
AI summary
The present disclosure relates to a method including reshaping a first tensor of weights, by using one or more second tensor having a lower dimension than the first tensor dimension and encoding the second tensor in a signal The present disclosure relates to a method including obtaining a first tensor of weights by reshaping one or more second tensor hav ing a lower dimension than the first tensor dimension, the one or more second tensor being decoded from a signal. The present disclosure further relates to the corresponding dev ices, signal, and computer readable storage media.