Neural Network Node Pruning with Bypass-Preserving Layer Pairing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing node pruning methods are inadequate for handling non-uniform and complex neural networks with non-linear activation functions, leading to inefficiencies in model size reduction and performance degradation.
Innovation Solution
A node pruning device and method that employs node entropy, inter-layer pairing, and layer-based pruning rates, while maintaining bypass connections to handle non-uniform networks effectively, using ReLU activation functions and specific threshold settings.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional node pruning methods are applied to simple neural networks (DNN, CNN), then model size reduction is achieved, but the method fails to handle non-uniform and complicated networks with bypass connections and non-linear activation functions
Solution Approach 1:
The patent segments the node pruning process into multiple independent modules: node activation section (for scoring), inter-layer pairing section (for handling connections), bypass setting section (for preserving bypass connections), and pruning execution section (for actual pruning). This segmentation allows each module to specialize in handling specific aspects of complex network topologies, making the overall system adaptable to non-uniform networks while maintaining performance through coordinated operation of these segments.
Solution Approach 2:
The patent applies different pruning strategies to different parts of the network based on their specific characteristics. The bypass setting section identifies and preserves bypass connections (local feature), while the pruning execution section applies layer-specific pruning rates (local quality). The node activation section uses different scoring metrics (node entropy, frequency-based node activity, variance-based node activity) for different node types. This localized adaptation allows the system to handle non-uniform network structures effectively.
2Productivity
If node pruning is performed to reduce model size, then calculation amount is reduced, but performance may deteriorate due to improper handling of complex network topologies
Solution Approach 1:
The patent performs preliminary actions before actual pruning: first calculating node scores using multiple metrics (node entropy, frequency-based node activity, variance-based node activity), then identifying inter-layer pairing relationships, and finally determining bypass connections. These preliminary analyses enable the pruning execution section to make informed decisions about which nodes to prune, ensuring that performance-critical nodes are preserved while still achieving significant model size reduction and calculation efficiency improvement.
Solution Approach 2:
The patent incorporates feedback mechanisms through the node activation section that continuously evaluates node importance using multiple scoring criteria. The system monitors network structure changes during pruning and adjusts the pruning strategy accordingly. The bypass setting section provides feedback by identifying connections that should be preserved, allowing the pruning execution section to modify its decisions to maintain model performance while still achieving computational efficiency.
3Adaptability or versatility
If sigmoid function is used as activation function, then conventional node pruning methods work, but recent models use non-linear functions like ReLU which require different handling
Solution Approach 1:
The patent creates a universal node pruning system that can handle multiple activation functions (sigmoid, ReLU, and others) through a unified framework. The node activation section uses different scoring metrics that are applicable to various activation functions: node entropy works for probability outputs, frequency-based node activity handles sparse outputs, and variance-based node activity works for any continuous output. This multi-functional approach allows the same pruning system to adapt to different activation functions without requiring separate specialized mechanisms for each function type.
Solution Approach 2:
The patent changes the parameters of the node scoring system to accommodate different activation functions. Instead of using a fixed scoring method, the system adjusts the scoring parameters based on the activation function characteristics. For ReLU functions, the system uses frequency-based and variance-based metrics that capture the sparsity and distribution characteristics of ReLU outputs. For sigmoid functions, node entropy remains effective. This parameter adaptation allows the pruning mechanism to maintain effectiveness across different activation functions without increasing fundamental system complexity.
Data Source
AI summary
A node pruning device for a network model in which a plurality of layers are continuously connected includes: a node activation section configured to select a node to be pruned on the basis of a score function that represents importance of a node; an inter-layer pairing section configured to prune an input connected to a node pruned at an output of a previous layer; a bypass setting section configured to provide a bypass connection between an input and an output of a layer and not to prune the bypass connection; and a pruning execution section configured to prune the nodes with the same pruning rate for each layer.