Multi-iteration Neural Network Compression via Sensitivity-Based Pruning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks, particularly those used in speech recognition, face challenges in scalability due to their large size, leading to high computational and storage demands, which restricts their deployment on mobile terminals and requires effective compression methods without compromising accuracy.
Innovation Solution
A multi-iteration compression method for deep neural networks that involves sensitivity analysis, pruning, and fine-tuning to determine and adjust initial compression ratios for matrices, allowing for the conversion of dense networks into sparse networks, thereby reducing storage and computational requirements while maintaining accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep neural networks are used for speech recognition, then recognition accuracy is improved, but computational resources and storage requirements increase
Solution Approach 1:
The patent extracts and removes redundant connections (edges) from the neural network graph based on importance metrics. By identifying and eliminating less important connections while preserving critical ones, the network achieves compression without significant accuracy loss, directly addressing the storage requirement issue while maintaining recognition performance
Solution Approach 2:
The patent applies different compression strategies to different parts of the network based on local importance characteristics. By computing connection importance metrics and applying selective pruning to specific regions or layers, the method optimizes the balance between compression ratio and accuracy preservation locally, rather than applying uniform compression across the entire network
2Measurement precision
If deep neural networks are used for speech recognition, then recognition accuracy is improved, but computational resources increase
Solution Approach 1:
The patent removes computationally expensive redundant connections from the network based on importance metrics. By eliminating less important edges and operations, the total computational workload is reduced, leading to lower energy consumption and faster inference while preserving the most critical computational paths that maintain accuracy
Solution Approach 2:
The patent changes the network architecture parameters by modifying the connection matrix sparsity pattern. Through iterative pruning that adjusts the density and distribution of remaining connections, the computational complexity parameters are optimized to reduce resource usage while maintaining the functional accuracy of the speech recognition system
3Quantity of substance
If compression methods are applied to neural networks, then storage and computational requirements are reduced, but accuracy may be compromised
Solution Approach 1:
The patent implements an iterative feedback mechanism where the network is pruned, evaluated for accuracy performance, and then further refined based on the evaluation results. This closed-loop process allows the system to adaptively adjust the compression level and connection selection to achieve the maximum possible compression while maintaining accuracy within acceptable thresholds
Solution Approach 2:
The patent performs preliminary identification and marking of important connections before actual pruning occurs. By pre-computing importance metrics and selecting which connections to preserve in advance, the method ensures that critical accuracy-maintaining pathways are protected during the compression process, preventing accuracy degradation
4Quantity of substance
If multi-iteration compression is applied to neural networks, then compression ratio is improved, but training time increases
Solution Approach 1:
The patent performs preliminary computation of connection importance metrics using efficient approximation methods before the actual iterative pruning begins. By pre-processing and pre-identifying redundant connections in the initial iterations, the method reduces the computational burden of subsequent refinement stages, achieving high compression ratios without linearly increasing total training time
Solution Approach 2:
The patent applies partial pruning in early iterations, focusing on removing obviously redundant connections with high confidence. By addressing the most obvious compression opportunities first with less computationally intensive methods, and then progressively refining with more sophisticated analysis, the system achieves diminishing returns optimization that balances compression ratio gains against increasing time costs
Data Source
AI summary
A multi-iteration method for compressing a deep neural network into a sparse neural network without degrading the accuracy is disclosed herein. In an example, the method includes determining a respective initial compression ratio for each of a plurality of matrices characterizing the weights between the neurons of the neural network, compressing each of the plurality of matrices based on the respective initial compression ratio, so as to obtain a compressed neural network, and fine-tuning the compressed neural network.