Neural Network Pruning via Weight Subset Decomposition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network pruning methods are limited by their dependence on specific architectures, heuristic criteria, and the need for training data, which can violate privacy regulations and require tuning of hyper-parameters, and they do not efficiently reduce computational resources on edge devices.
Innovation Solution
A method that decomposes sub-neurons into subsets with equal weights, removes redundant subsets, and replaces their outputs with substitutes, allowing for data-free, parameter-free pruning that reduces computational resources without altering the network's outputs, thus improving real-time performance on edge devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If neural network pruning is performed using existing methods, then computational requirements are reduced, but the methods depend on specific architectures and heuristic criteria without theoretical evidence of efficiency
Solution Approach 1:
The patent changes the parameter being optimized from heuristic criteria (e.g., magnitude-based pruning) to information-theoretic entropy of weight distributions. By using entropy as the pruning criterion, the method achieves architecture-agnostic pruning with theoretical guarantees, resolving the contradiction between computational efficiency and architecture dependency.
Solution Approach 2:
The patent uses hashing to create compressed representations (copies) of weight distribution characteristics. Instead of working with full weight matrices, it computes hash values that capture essential distribution properties, enabling efficient pruning decisions without architecture-specific computations.
2Productivity
If data-driven pruning methods are used, then pruning effectiveness is improved, but privacy regulations such as GDPR may be violated due to use of training data
Solution Approach 1:
The patent extracts the essential information needed for pruning (weight distribution characteristics) and represents it through hashing. This extraction allows the method to achieve data-driven pruning effectiveness while removing the need to store or process actual training data, thus complying with privacy regulations.
Solution Approach 2:
The patent substitutes the mechanical process of storing and processing training data with an information-theoretic approach using entropy and hashing. This substitution maintains pruning effectiveness while eliminating privacy violations by never requiring access to actual training samples.
3Manufacturing precision
If multiple hyper-parameters are used in pruning algorithms, then pruning precision is improved, but the algorithms become hard to tune
Solution Approach 1:
The patent creates a universal pruning criterion based on entropy that works across different architectures and scenarios without requiring architecture-specific hyper-parameters. The entropy-based approach serves multiple functions (pruning decision, compression ratio control, accuracy preservation) through a single theoretical framework, eliminating the need for extensive hyper-parameter tuning.
4Reliability
If computation power is increased to maintain DNN performance, then predictive functionality is improved, but deployment on edge devices becomes limited
Solution Approach 1:
The patent segments the neural network by identifying and removing redundant neurons based on entropy analysis. This segmentation reduces the network size and computation requirements while preserving the essential predictive functionality, enabling deployment on resource-constrained edge devices.
Solution Approach 2:
The patent performs pruning as a preliminary action before deployment on edge devices. By using entropy-based analysis to remove redundant computations in advance, the network is optimized for low-power execution while maintaining its predictive capabilities during inference on edge devices.
Data Source
Figure 1~2
Figure 3(A)~3(C)
Figure 4
AI summary
The disclosure relates to a computer-implemented method for pruning a neural network which comprises one or more layers each comprising one or more input channels and one or more output channels, each of the one or more layers comprising one or more sub-neurons each comprising one or more respective weights, each sub-neuron outputting an intermediate output. The method comprises: providing a trained neural network, for each layer of neural network, decomposing the one or more sub-neurons into one or more first subsets and one or more second subsets each associated to a first subset, wherein a value of the weight(s) of sub-neuron(s) in each second subset is substantially equal to a value of weights of the respective sub-neurons in the associated first subset, and for each of the one or more second subsets, splitting the trained neural network by removing the second subset of the sub-neurons from the layer; and replacing an intermediate output of the second subset by a substitute of an intermediate output of the associated first subset.