Second-Order Neural Network Pruning for Accuracy-Preserving Simplification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing pruning methods for neural networks often result in diminished total capacity or accuracy, and fail to predict how pruning impacts the remaining network structures, making it difficult to simplify neural networks without adversely affecting their performance.
Innovation Solution
A method involving the calculation of a product of a matrix with partial second-order derivations of a loss function and pruning vectors to determine changes in the loss function, allowing for precise pruning of neural network structures to generate a simplified neural network with minimal impact on capacity or accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If pruning methods are applied to reduce neural network size, then device complexity is reduced, but total capacity or accuracy is diminished
Solution Approach 1:
The patent calculates the product of the Hessian matrix and pruning vectors before actually performing the pruning operation. This preliminary calculation allows the system to predict the impact of pruning on loss function changes, enabling informed decisions about which structures to prune while maintaining accuracy, thus resolving the contradiction between reducing complexity and preserving reliability
Solution Approach 2:
The patent uses the calculated product of the Hessian matrix and pruning vectors as feedback to determine the actual changes in the loss function. This feedback mechanism allows the system to monitor and adjust the pruning process, ensuring that only structures whose removal causes acceptable loss function changes are pruned, thereby maintaining accuracy while reducing complexity
2Device complexity
If multiple network structures are pruned simultaneously, then device complexity is reduced, but the impact on total capacity cannot be predicted
Solution Approach 1:
The patent performs preliminary calculations by computing the product of the Hessian matrix and pruning vectors for multiple network structures before executing the pruning. This allows the system to predict the combined impact on the loss function, making the capacity impact measurable and predictable, thus resolving the contradiction between reducing the number of structures and maintaining predictability
Solution Approach 2:
The patent introduces the Hessian matrix as an intermediary that captures the relationships between multiple network structures and the loss function. By calculating the product of the Hessian matrix and pruning vectors, the system creates a measurable intermediate representation that predicts the impact of pruning multiple structures simultaneously, resolving the difficulty of detecting and measuring capacity changes
Data Source
AI summary
A method for generating a simplified computer-implemented neural network. The method includes: receiving a predefined neural network, which includes a plurality of neural network structures and is described by weights, each neural network structure being assigned a pruning vector which describes a change in weights as a result of the pruning of the respective neural network; calculating a product of a matrix including a structure vector, the matrix including partial second order derivations of a loss function with respect to the plurality of weights; determining changes in the loss function with respect to the predefined neural network, each change occurring as a result of a pruning of a corresponding neural network structure of the two or more neural network structures to be pruned; and pruning at least one neural network structure based on the determined two or more changes in the loss function to generate the simplified neural network.