Deep Network Layer Rank Determination for Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks require significant computational resources, making them unsuitable for resource-constrained platforms like mobile devices and IoT devices, as they are computationally intensive and challenging to train.
Innovation Solution
The method involves determining a low-rank approximation for layers of the neural network using singular value decomposition (SVD) without fine-tuning, allowing for compression based on defined residual targets, which reduces model size and computational complexity while maintaining accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep neural networks are used to achieve excellent classification results, then accuracy is improved, but computational complexity increases prohibitively
Solution Approach 1:
The patent segments the weight matrices of neural network layers into multiple low-rank components using tensor decomposition. By representing weights as a sum of rank-1 outer products, the network is divided into multiple computational paths that can be processed separately and combined, reducing the overall computational burden while preserving classification accuracy.
Solution Approach 2:
The patent changes the parameter representation of weight matrices from full-rank to low-rank factorizations. By transforming the weight parameters into decomposed forms with fewer effective parameters, the computational complexity is reduced while maintaining the network's ability to achieve excellent classification results on benchmarks.
2Productivity
If larger computational networks are used to process more data, then processing capability is improved, but resource consumption increases
Solution Approach 1:
The patent introduces dynamic routing mechanisms that selectively activate different computational paths based on input characteristics. The dynamic nature of the decomposed network allows it to adaptively allocate computational resources, processing complex patterns when necessary while consuming fewer resources for simpler inputs, thus improving processing capability without proportionally increasing resource consumption.
Solution Approach 2:
By segmenting the network into multiple low-rank computational paths, the system can selectively engage only the necessary paths for each input, rather than always executing the full network capacity. This selective activation improves processing capability for diverse inputs while reducing average resource consumption.
3Ease of operation
If network compression is applied to reduce model size, then device suitability is improved, but training difficulty increases
Solution Approach 1:
The patent performs network compression through low-rank factorization during the network design phase rather than requiring post-training fine-tuning. By pre-computing the decomposed weight representations and integrating them into the network architecture beforehand, the compressed network can be deployed on resource-constrained devices without encountering the training difficulties that would arise from attempting to fine-tune compressed networks later.
Data Source
AI summary
An apparatus of operating a computational network is configured to determine a low-rank approximation for one or more layers of the computational network based at least in part on a set of residual targets. A set of candidate rank vectors corresponding to the set of residual targets may be determined. Each of the candidate rank vectors may be evaluated using an objective function. A candidate rank vector may be selected and used to determine the low rank approximation. The computational network may be compressed based on the low-rank approximation. In turn the computational network may be operated using the one or more compressed layers.