Neural Network Pruning Thresholds

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Deep learning neural networks, particularly convolutional neural networks, face challenges with high computational loads and power consumption due to a large number of parameters, and existing pruning methods struggle to set optimal thresholds for reducing parameters without compromising performance, especially when multiple layers are involved.

Innovation Solution

The method involves automatically determining thresholds for pruning neural network layers and repeating the pruning process until a pruning error allowance is reached, allowing for the reduction of parameters while maintaining performance through retraining with the remaining weights.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Use of energy by moving object

If the number of parameters in neural network is reduced through pruning, then computational load and power consumption are decreased, but pruning performance and accuracy may deteriorate

Engineering Contradiction:

Improvepower consumptionVSAvoidpruning performance

Core Design Contradiction:

Use of energy by moving objectVSReliability

Solution Approach 1:

The patent dynamically adjusts the pruning threshold parameter based on layer-specific characteristics and error allowances. Different thresholds are applied to different layers, and the threshold is iteratively refined until the pruning error reaches the allowance, optimizing the balance between parameter reduction and performance maintenance

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The pruning process is made dynamic through iterative adjustment of thresholds and error allowances. The system repeatedly prunes with different thresholds and evaluates pruning errors, adapting the pruning strategy based on actual performance degradation, rather than using a fixed static threshold

Inventive Principle:

Principle #15Dynamics

2Ease of manufacture

If a fixed threshold is used for pruning all layers, then the pruning process is simple, but it cannot accommodate different error allowances of different layers

Engineering Contradiction:

Improvepruning process simplicityVSAvoidlayer-specific error allowance adaptation

Core Design Contradiction:

Ease of manufactureVSAdaptability or versatility

Solution Approach 1:

The patent applies different pruning thresholds to different layers based on their specific error allowances. Each layer is evaluated individually, and the threshold is adjusted according to the layer's tolerance for pruning error, allowing fine-grained control of the pruning process across the network architecture

Inventive Principle:

Principle #3Local quality

3Reliability

If multiple pruning iterations with different thresholds are performed, then optimal pruning is achieved, but computational time and complexity increase

Engineering Contradiction:

Improvepruning accuracyVSAvoidpruning time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system incorporates feedback through iterative evaluation of pruning errors. After each pruning iteration, the pruning error is calculated and compared against the allowance, and the threshold is adjusted accordingly. This feedback mechanism ensures optimal pruning while stopping when the error allowance is satisfied, avoiding unnecessary additional iterations

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS12008474B2Automatic thresholds for neural network pruning and retraining

Publication Date: 2024.06.11 SAMSUNG ELECTRONICS CO LTD

AI summary

An embodiment includes a method, comprising: pruning a layer of a neural network having multiple layers using a threshold; and repeating the pruning of the layer of the neural network using a different threshold until a pruning error of the pruned layer reaches a pruning error allowance.