Artificial Neural Network Growth via Weight Threshold Pruning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
State-of-the-art artificial neural networks are over-built, leading to excessive resource consumption and inefficiency, as many connections can be pruned without affecting accuracy, particularly in fully connected layers.
Innovation Solution
An artificial neural network is grown from a seed network by analyzing training data to determine valuable connections based on neuron activity, focusing on connections with outputs one standard deviation above and below the average, reducing the network size and computational power requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a fully connected layer is built with all possible connections between neurons, then the network coverage and potential accuracy are maximized, but the number of connections and computational resources required increase significantly
Solution Approach 1:
The patent extracts and removes unnecessary connections from the fully connected layer by analyzing weight magnitudes. Connections with weights below a threshold (near-zero weights) are identified and eliminated, retaining only the significant connections that contribute meaningfully to the output. This extraction process reduces the number of connections while preserving network accuracy.
Solution Approach 2:
Instead of implementing all possible connections in a fully connected layer, the patent applies partial action by selectively establishing only the necessary connections based on weight analysis. This approach avoids the excessive action of creating all possible connections, thereby reducing complexity while maintaining sufficient accuracy for the task.
2Productivity
If the number of connections in the network is reduced through pruning, then execution time and resource consumption decrease, but network accuracy may be compromised
Solution Approach 1:
The patent employs feedback by analyzing the weights of connections during or after training to determine which connections are necessary and which can be removed. This feedback mechanism guides the pruning process, ensuring that connections critical for accuracy are preserved while removing only those with minimal impact, thus maintaining accuracy while improving execution speed.
Solution Approach 2:
The patent changes the parameter of connection significance by introducing a weight threshold. Connections are evaluated based on their weight values, and only those exceeding the threshold are retained. This parameter-based selection ensures that pruning does not compromise accuracy, as only connections with sufficiently large weights (and thus significant impact on output) are preserved.
3Ease of manufacture
If traditional full connection methods are used in fully connected layers, then implementation simplicity is maintained, but memory usage and power consumption increase
Solution Approach 1:
The patent extracts and removes unnecessary connections from the fully connected layer by analyzing weight magnitudes. Connections with weights below a threshold (near-zero weights) are identified and eliminated, retaining only the significant connections that contribute meaningfully to the output. This extraction process reduces the number of connections while preserving network accuracy.
Solution Approach 2:
The patent changes the parameter of connection significance by introducing a weight threshold. Connections are evaluated based on their weight values, and only those exceeding the threshold are retained. This parameter-based selection ensures that pruning does not compromise accuracy, as only connections with sufficiently large weights (and thus significant impact on output) are preserved.
Data Source
AI summary
A method to grow an artificial neural network is disclosed. A seed neural network is trained on all classes in a dataset. All classes in the dataset are applied to the seed network, and average output values of the seed network are calculated. Class members that are nearest to and furthest from the average output values are selected, the class members are applied to the seed network, and a standard deviation is calculated. Perceptrons are added to the seed network, and inputs of the added perceptrons are connected to the seed layer based on the calculated standard deviation. A classifier is then added to the outputs of the added perceptrons, and the seed network and the added perceptrons are trained using all members in the dataset.