Neural Network Pruning Across Layers for Resource-Limited AI
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computer-implemented neural networks require significant hardware resources and computational power due to their complex structures, making them unsuitable for devices with limited resources or requiring rapid response times.
Innovation Solution
A method to generate a simplified neural network by pruning neural network substructures while considering the changes in the loss function using a matrix that accounts for correlations between substructures across different layers, reducing the network size without significantly impacting performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a complex neural network structure is used to maintain high performance and accuracy, then the performance capability is improved, but the hardware resource requirements and device complexity increase excessively
Solution Approach 1:
The patent segments the neural network into individual substructures (neurons, layers, connections) that can be independently evaluated and selectively removed. The Hessian matrix is divided into submatrices corresponding to different substructures, enabling granular pruning decisions that maintain overall network performance while reducing complexity.
Solution Approach 2:
The patent extracts and removes redundant or less important neural network substructures based on the pruning priority values derived from the Hessian matrix analysis. This extraction process eliminates unnecessary components while preserving the essential functionality and performance of the network.
2Device complexity
If neural network substructures are pruned to reduce size and complexity, then device complexity is reduced, but the performance and accuracy of the network deteriorate
Solution Approach 1:
The patent uses the Hessian matrix to provide feedback information about the impact of pruning each substructure on the loss function. The pruning priority values indicate how much each substructure contributes to the overall performance, enabling selective removal of substructures with minimal impact while maintaining network accuracy.
Solution Approach 2:
The patent changes the parameter of network complexity by selectively removing substructures based on their pruning priority values. This parameter change is guided by the Hessian matrix analysis, which identifies which substructures can be removed with minimal impact on performance, thus reducing complexity while preserving accuracy.
3Quantity of substance
If traditional pruning methods are used to reduce network size, then the number of neurons and connections is reduced, but the assessment time and computational overhead increase due to not considering correlations between substructures
Solution Approach 1:
The patent merges the evaluation of multiple substructures by computing the Hessian matrix that captures correlations between different neural network substructures. This combined evaluation allows simultaneous consideration of multiple substructures' impacts on the loss function, enabling more efficient pruning decisions that reduce assessment time compared to evaluating each substructure independently.
Solution Approach 2:
The patent performs preliminary computation of the Hessian matrix and pruning priority values before actual pruning occurs. This preliminary action pre-calculates the impact of pruning each substructure, allowing for rapid pruning decisions without repeated expensive evaluations during the pruning process itself.
Data Source
AI summary
Methods for generating a simplified computer-implemented neural network for a device. The method includes receiving a predefined neural network, which includes a plurality of neural network substructures in two or more layers; computing an approximative neural network for the predefined neural network, and computing a matrix with respect to the approximative neural network, which describes changes in the loss function due to simultaneous pruning of at least two network substructures from the plurality of neural network substructures which are situated within different layers of the two or more layers of the predefined neural network. The method finally includes pruning, based on the changes in the loss function described in the matrix, at least one neural network substructure from the plurality of neural network substructures to generate the simplified neural network.