Neural Network Weight Clustering for Execution Time
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Artificial neural networks require significant execution time and computational resources, making them unsuitable for embedded systems with limited resources, and existing solutions like pruning algorithms are costly and limit network complexity.
Innovation Solution
A method to simplify trained neural networks by forming clusters of weights and replacing them with representative weights, which are computed to minimize the cost function, allowing for faster execution without retraining.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a trained neural network is used with full complexity, then accuracy is maintained, but execution time is high and computational resources are excessive
Solution Approach 1:
The patent merges multiple weight values into cluster representatives. Weights that are similar in value are grouped together and replaced by a single representative weight, reducing the total number of parameters while preserving the overall functionality and accuracy of the network.
Solution Approach 2:
The patent changes the parameter representation from individual weight values to clustered representative weights. By transforming the weight space into clusters and using centroid or representative values, the network maintains its computational accuracy while reducing the number of parameters that need to be stored and processed.
2Productivity
If pruning algorithms are applied to reduce network complexity, then execution time decreases, but retraining is required which is lengthy and costly
Solution Approach 1:
The patent performs weight clustering as a preliminary action after training, before deployment. This preprocessing step organizes the weights into clusters and replaces them with representative values, creating an optimized network structure that can be executed faster without requiring any retraining phase.
Solution Approach 2:
The patent creates a simplified copy of the trained network by replacing individual weights with cluster representative weights. This copied structure maintains the network's predictive capabilities while reducing complexity, allowing direct deployment without retraining.
3Productivity
If fewer neurons and hidden layers are used, then execution time is reduced, but the network's ability to solve complex problems is limited
Solution Approach 1:
The patent applies local quality by making different weights have the same value within their cluster while allowing different clusters to have different representative weights. This enables the network to maintain complexity where needed (different representative weights for different functional regions) while reducing redundancy (same weights within clusters).
Data Source
AI summary
According to one aspect, it is proposed a method for simplifying a trained artificial neural network, the method including: obtaining a trained neural network having layers of neurons, each layer being configured to receive at least one input, each input being connected to at least one neuron of the layer by a connection applying a weight, named trained weight, to the input, and for each input of each layer of the trained neural network: ∘ forming clusters of trained weights of the connections of the layer connected to said input of the layer, ∘ computing a representative weight for each cluster, the representative weight being representative of the weights of the cluster, ∘ replacing in the trained neural network the trained weights of each cluster by the representative weight of this cluster to obtain a simplified neural network.