Neural Network Model Update via Layered Weight Quantization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network models are complex and require high computing power, making them unsuitable for devices with limited resources, and traditional quantization methods introduce errors that affect performance.
Innovation Solution
A method and electronic device that updates neural network models by quantizing neuron weights and performing model order-reduction, generating new neurons and updating activation functions based on estimated values to maintain performance while reducing complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If traditional quantization method is used to quantize the neural network model, then the model complexity is reduced, but the performance of the quantized neural network model is affected by quantization error accumulated layer by layer
Solution Approach 1:
The patent implements a feedback mechanism where the system evaluates the performance impact of quantization at each layer and dynamically adjusts the quantization strategy. By monitoring the accumulated quantization errors and using this information to refine subsequent quantization decisions, the system maintains model performance while achieving complexity reduction through iterative optimization.
Solution Approach 2:
The patent changes the quantization parameters dynamically based on the specific layer and its importance in the neural network. Different layers are assigned different quantization precision levels, and the quantization bits are adjusted according to the layer's contribution to overall model performance, allowing critical layers to maintain higher precision while less critical layers use lower precision for complexity reduction.
2Reliability
If heuristic methods are used to generate neural network models, then the model performance is improved, but the model complexity is not effectively reduced
Solution Approach 1:
The patent segments the neural network model into different layers and applies targeted quantization strategies to each segment. By dividing the model into critical and non-critical layers, the system can maintain high performance in critical segments while reducing complexity in non-critical segments, achieving overall model simplification without sacrificing essential functionality.
Solution Approach 2:
The patent applies different quantization quality levels to different parts of the neural network model based on their local importance. Critical layers that have greater impact on model performance receive higher quantization quality, while less important layers use lower quality quantization, creating a non-uniform quality distribution that optimizes the trade-off between performance and complexity.
3Reliability
If neural network models are designed for high computing power devices, then the model performance is maintained, but the models cannot run smoothly on devices with limited computing power
Solution Approach 1:
The patent creates a universal quantization framework that can be applied across different device types and computing power levels. The system generates quantized models that are adaptable to various hardware configurations, allowing the same base model to be efficiently deployed on both high-performance and resource-constrained devices through systematic quantization and optimization techniques.
Data Source
AI summary
A method and an electronic device of updating a neural network model are provided. The method includes the following steps. The neural network model is received. The neural network model includes a first neuron and a second neuron. The training data is input to the first neuron to output a first estimated value from the second neuron. A first weight of the first neuron is quantized to generate a third neuron, and a second weight of the second neuron is quantized to generate a fourth neuron. The training data is input to the third neuron to output a second estimated value from the fourth neuron. A first activation function of the first neuron and a second activation function of the second neuron are updated according to the first estimated value and the second estimated value, thereby generating the updated neural network model. The updated neural network model is output.