Low Displacement Rank Approximation for DNN Weight Matrix Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep Neural Networks (DNNs) face high inference complexity due to their massive number of parameters, making it challenging to bring their performance to mobile or embedded devices with resource limitations on battery size, computational power, and memory capacity.
Innovation Solution
The method involves compressing pre-trained DNNs using Low Displacement Rank (LDR) approximation, which decomposes layer weight matrices into a sum of structured matrices, enabling simultaneous compression and reducing inference complexity, and includes a syntax structure for encoding and decoding LDR layers in a bitstream.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If DNNs use a huge number of parameters to achieve state of the art performance, then performance is improved, but inference complexity increases prohibitively
Solution Approach 1:
The patent segments the weight matrices of DNN layers into multiple smaller matrices (e.g., dividing a large weight matrix into four smaller matrices). This segmentation reduces the computational complexity of inference operations while maintaining the representational capacity needed for high performance. The segmented matrices can be processed in smaller batches, reducing the burden on mobile and embedded devices.
Solution Approach 2:
The patent employs a nested structure where multiple low-rank matrices are combined to represent the original high-dimensional weight matrices. The decomposition approach nests smaller matrices within a hierarchical structure that reconstructs the functionality of the original large matrices, enabling efficient computation while preserving performance.
2Measurement precision
If DNNs have massive number of parameters, then performance is improved, but storage requirements increase
Solution Approach 1:
The patent extracts only the essential components needed for DNN functionality by decomposing weight matrices into low-rank representations. Instead of storing all parameters of the original matrices, the method extracts and stores only the critical low-rank factors, significantly reducing storage requirements while maintaining the ability to reconstruct the necessary computational functionality.
Solution Approach 2:
The patent changes the parameter representation from full-rank weight matrices to low-rank decomposed matrices. By altering the mathematical representation of the parameters (from dense to sparse/low-rank form), the system achieves the same performance with fewer stored parameters, directly addressing the storage requirement challenge.
3Adaptability or versatility
If DNNs are deployed on mobile or embedded devices, then accessibility is improved, but resource limitations on battery size, computational power, and memory capacity are exceeded
Solution Approach 1:
The patent introduces dynamic adaptability by enabling DNNs to operate with variable precision and decomposition levels. The system can dynamically adjust the level of matrix decomposition and computation precision based on available resources, allowing the same model to run efficiently on devices with different computational power, memory capacity, and battery constraints.
Solution Approach 2:
The patent changes the computational parameters by using low-rank approximations that reduce both computational operations and memory access requirements. This parameter transformation enables the DNN to run on resource-constrained mobile and embedded devices by reducing the energy consumption associated with matrix multiplications and parameter storage.
Data Source
AI summary
A method and apparatus for conveying information in a bitstream for deep neural network compression, such as in matrices representing weights, biases and non-linearities, to iteratively compress a pre-trained deep neural network by low displacement rank based approximation of the network layer weight matrices. The low displacement rank approximation allows for replacement of an original layer weight matrices of the pre-trained deep neural network as the sum of small number of structured matrices, allowing compression and low inference complexity. A decoder stage parses a bitstream for inference.