Neural Network Model Compaction via Selective Node Removal
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional artificial neural networks (ANNs) are large and resource-intensive, consuming significant storage space, bandwidth, and processing resources due to their numerous nodes and connections, and existing methods to reduce their size do not effectively compact internal hidden layers without sacrificing accuracy.
Innovation Solution
The method involves selectively deactivating and removing nodes from ANNs during training based on their contribution to the network's accuracy, adjusting activation probabilities towards zero or one, and permanently removing nodes that do not significantly impact the network's performance, resulting in a more compact network without accuracy loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional ANNs are used with all nodes active, then accuracy is maintained, but storage space and processing resources are excessively consumed
Solution Approach 1:
The patent extracts and removes redundant nodes from the neural network that do not contribute significantly to accuracy. By identifying nodes with minimal impact on model performance and permanently deactivating them, the network size is reduced while maintaining accuracy, directly resolving the contradiction between storage efficiency and reliability
Solution Approach 2:
The patent changes the activation probability parameter of nodes from a uniform distribution to a selective distribution based on their contribution to accuracy. Nodes are assigned activation probabilities of either 0 or 1 based on their importance, transforming the network from a dense structure to a sparse one that maintains accuracy while reducing storage requirements
2Reliability
If conventional ANNs are used with all nodes active, then model completeness is maintained, but bandwidth usage increases
Solution Approach 1:
The patent removes redundant nodes that do not contribute to model completeness or accuracy. By extracting and eliminating these unnecessary components, the network transmits less data during inference, reducing bandwidth consumption while preserving the essential functional completeness of the model
3Reliability
If conventional ANNs are used with all nodes active, then computational thoroughness is maintained, but processing resources are excessively consumed
Solution Approach 1:
The patent extracts and removes nodes that do not contribute significantly to computational thoroughness. By eliminating these redundant computational units, the network performs fewer unnecessary calculations during inference, reducing processing resource consumption while maintaining the thoroughness needed for accurate predictions
Solution Approach 2:
The patent applies partial activation to nodes based on their importance. Instead of activating all nodes uniformly, only essential nodes with activation probability of 1 are fully activated, while redundant nodes with activation probability of 0 are completely deactivated. This partial action approach maintains computational thoroughness for critical paths while eliminating waste on redundant computations
4Productivity
If node removal is performed to compact the network, then storage and processing efficiency improve, but accuracy may be sacrificed
Solution Approach 1:
The patent uses feedback from accuracy measurements to guide the node removal process. During training, the system monitors accuracy changes as nodes are deactivated and adjusts the selection of nodes for removal accordingly. This feedback mechanism ensures that nodes critical to accuracy are preserved while redundant nodes are removed, achieving compaction without sacrificing performance
Solution Approach 2:
The neural network performs self-pruning by automatically identifying and removing its own redundant nodes during the training process. The network evaluates the contribution of each node and autonomously decides which nodes to deactivate, eliminating the need for external intervention or manual selection while maintaining accuracy through self-regulated optimization
Data Source
AI summary
The present disclosure is directed to the generation of a compact artificial neural network by removing individual nodes from the artificial neural network. Individual nodes of the artificial neural network may be deactivated randomly and/or selectively during training of the artificial neural network. In some embodiments, a particular node may be randomly deactivated approximately half of the time during processing of a set of training data inputs. Based on the accuracy of the results obtained when the node is deactivated compared to the accuracy of the results obtained when the node is activated, an activation probability may be generated. Nodes can then be selectively removed from the artificial neural network based on the activation probability.