Artificial Neural Network Simplification via Dual Assessment Pruning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Artificial neural networks (ANNs) require significant computing power and energy for training, leading to inefficiencies and a tendency towards 'overfitting', where they memorize training data rather than understanding the underlying knowledge, and they often require substantial memory resources.
Innovation Solution
A method is introduced to simplify ANNs by using two complementary assessment criteria to identify and set to zero or deactivate parameters and neurons, optimizing the configuration for better generalization and reduced complexity, which can be applied before or after training, thereby reducing computing time and memory demands.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the ANN is trained with all parameters to achieve high accuracy, then the classification performance is improved, but the computing time and energy consumption increase significantly
Solution Approach 1:
The patent applies preliminary action by performing pruning assessment and parameter identification before the main training process. The method identifies and sets to zero certain parameters (such as weights with very small gradients or those identified through sensitivity analysis) before training begins, thereby reducing the computational burden during training while preserving the network's ability to achieve high accuracy on important features.
2Measurement precision
If the ANN is trained with all parameters to achieve high accuracy, then the classification performance is improved, but the energy consumption increases significantly
Solution Approach 1:
The patent applies the taking out principle by extracting and removing unnecessary parameters from the neural network before training. Through assessment criteria that evaluate parameter importance (such as gradient magnitude analysis or sensitivity measures), the method identifies and removes parameters that contribute minimally to the output, thereby reducing energy consumption during training and inference while maintaining classification accuracy.
3Measurement precision
If the ANN memorizes training data through overfitting, then the training accuracy is improved, but the generalization ability to unseen data deteriorates
Solution Approach 1:
The patent applies preliminary anti-action by implementing pruning measures before training to prevent overfitting. By identifying and zeroing out parameters that are likely to cause memorization of training data (such as those with extremely small but non-zero gradients that may capture noise), the method suppresses the tendency toward overfitting in advance, thereby improving generalization ability while maintaining adequate training accuracy.
4Productivity
If the ANN configuration is simplified by setting parameters to zero, then the computing resources required are reduced, but the classification accuracy may deteriorate
Solution Approach 1:
The patent applies local quality by selectively applying pruning to specific parameters based on their individual importance assessment. Rather than uniformly reducing the network, the method identifies specific parameters (weights, biases, or even entire neurons) that can be removed without significantly impacting accuracy, while preserving parameters that are critical for classification performance. This localized approach maintains computing efficiency while protecting classification accuracy.
Data Source
AI summary
A method for simplifying an artificial neural network (ANN) whose behavior is characterized by trainable parameters. In the method: a first assessment criterion is provided, which maps simplified configurations of the ANN on predictions for the performance of the ANN in the particular configuration; a second assessment criterion is provided, which also maps simplified configurations of the ANN on predictions for the performance of the ANN in the particular configuration, this second assessment criterion being at least partially complementary to the first assessment criterion; a simplified configuration of the ANN is optimized with the goal that this simplified configuration is mapped both by the first assessment criterion and also by the second assessment criterion, and/or by an overall assessment criterion resulting from a combination of both assessment criteria, on the best possible prediction for the performance of the ANN.