Neural Network Weight Pruning for Lower Memory and Overfitting

Solution Overview

Problem

The increasing number of nodes and weighted connections in neural networks leads to high memory and processing resource demands, and overfitting issues due to unintentional biasing or excessive training, which affects the performance of object recognition systems.

Innovation Solution

A neural network is generated with lightened weighted connections by setting values equal to zero for connections with absolute values less than a predetermined threshold, converting to a 16-bit fixed-point representation, and applying regularization to reduce resource requirements and mitigate overfitting.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If the number of nodes and weighted connections in the neural network increases to improve recognition capability, then the recognition performance is improved, but the memory and processing resources required increase rapidly

Engineering Contradiction:

Improverecognition performanceVSAvoidmemory and processing resources

Core Design Contradiction:

Measurement precisionVSQuantity of substance

Solution Approach 1:

The patent extracts and removes unnecessary weighted connections from the neural network through the lightening process. By setting small-weight connections to zero and pruning redundant connections, the system retains only the essential connections needed for recognition, thereby reducing memory and processing resources while maintaining recognition performance.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent changes the parameters of weighted connections by applying a lightening function that transforms the weight values. Connections with weights below a certain threshold are set to zero, effectively changing the parameter distribution of the network to reduce the quantity of active parameters while preserving the essential recognition capabilities.

Inventive Principle:

Principle #35Parameter changes

2Measurement precision

If the number of nodes and weighted connections in the neural network increases to improve recognition capability, then the recognition performance is improved, but the device complexity increases

Engineering Contradiction:

Improverecognition performanceVSAvoidneural network structure complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent extracts and removes unnecessary weighted connections from the neural network through the lightening process. By setting small-weight connections to zero and pruning redundant connections, the system retains only the essential connections needed for recognition, thereby reducing memory and processing resources while maintaining recognition performance.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent changes the parameters of weighted connections by applying a lightening function that transforms the weight values. Connections with weights below a certain threshold are set to zero, effectively changing the parameter distribution of the network to reduce the quantity of active parameters while preserving the essential recognition capabilities.

Inventive Principle:

Principle #35Parameter changes

3Measurement precision

If excessive training is applied to improve recognition performance, then the recognition accuracy is improved, but overfitting occurs due to unintentional biasing

Engineering Contradiction:

Improverecognition accuracyVSAvoidgeneralization capability

Core Design Contradiction:

Measurement precisionVSReliability

Solution Approach 1:

The patent applies preliminary actions during the training process by incorporating the lightening function into the training algorithm. By progressively applying lightening to weighted connections during training, the system prevents overfitting before it occurs, allowing the network to learn essential patterns without becoming overly specialized to training data noise.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent changes the parameters of weighted connections by applying a lightening function that transforms the weight values. Connections with weights below a certain threshold are set to zero, effectively changing the parameter distribution of the network to reduce the quantity of active parameters while preserving the essential recognition capabilities.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS12561560B2Method and apparatus with neural network lightening

Publication Date: 2026.02.24 SAMSUNG ELECTRONICS CO LTD

AI summary

A lightened neural network method and apparatus. The neural network apparatus includes a processor configured to generate a neural network with a plurality of layers including plural nodes by applying lightened weighted connections between neighboring nodes in neighboring layers of the neural network to interpret input data applied to the neural network, wherein lightened weighted connections of at least one of the plurality of layers includes weighted connections that have values equal to zero for respective non-zero values whose absolute values are less than an absolute value of a non-zero value. The lightened weighted connections also include weighted connections that have values whose absolute values are no greater than an absolute value of another non-zero value, the lightened weighted connections being lightened weighted connections of trained final weighted connections of a trained neural network whose absolute maximum values are greater than the absolute value of the other non-zero value.