Neural Network Average Quantization for Even Weight Distribution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The accuracy of neural networks decreases during quantization due to the uneven distribution of weights, which is not addressed by existing methods.
Innovation Solution
A neural network construction method and apparatus that utilizes the Gini coefficient to even the weight distribution by incorporating it into a regularized loss function, enhancing the accuracy of quantization by adjusting weight distributions during training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If floating-point weights are quantized to integers for hardware implementation, then ease of operation and hardware compatibility are improved, but manufacturing precision and network accuracy deteriorate
Solution Approach 1:
The patent applies preliminary action by performing average quantization on the weight combinations before the final quantization process. The method calculates average values of weight combinations in advance, uses these averages to determine quantization parameters, and then performs the actual quantization. This preliminary preparation of average values helps maintain accuracy while enabling integer quantization for hardware implementation.
Solution Approach 2:
The patent changes the parameter representation from individual floating-point weights to average quantization parameters derived from weight combinations. By transforming the weights into average values that represent groups of weights, the method enables more efficient integer quantization while preserving the essential information needed for accurate network operation on embedded hardware.
2Device complexity
If standard quantization methods are used to convert floating-point weights to integers, then device complexity is reduced, but measurement precision and network accuracy deteriorate
Solution Approach 1:
The patent transforms the quantization parameters from individual weight values to average values of weight combinations. This parameter transformation allows for more accurate representation with fewer bits, improving quantization accuracy while maintaining reduced device complexity. The average values capture the essential characteristics of weight combinations more efficiently than individual weight quantization.
3Productivity
If weight distribution is not adjusted during training, then training speed is maintained, but manufacturing precision and quantization accuracy deteriorate
Solution Approach 1:
The patent implements feedback by calculating the Gini coefficient during training to monitor weight distribution uniformity. The Gini coefficient serves as a feedback signal that indicates whether the weight distribution needs adjustment. This feedback mechanism allows the training process to adaptively adjust weights to achieve more uniform distribution, improving quantization accuracy without significantly compromising training speed.
Solution Approach 2:
The patent performs preliminary adjustment of weight distribution during the training phase by incorporating Gini coefficient calculations and adjustments. By addressing weight distribution issues before the quantization stage, the method ensures that weights are properly distributed in advance, leading to better quantization results without requiring extensive post-processing or retraining.
Data Source
AI summary
The present invention discloses a neural network construction method having average quantization mechanism that includes steps outlined below. A weight combination included in each of network layers of a neural network is retrieved. A loss function is generated according to the weight combination of all the network layers and target values. Corresponding to each network layers, a Gini coefficient of the weight combination is calculated and the Gini coefficients corresponding to all the network layers are accumulated as a regularized correction term. The loss function and the regularized correction term are merged as a regularized loss function to perform training on the neural network accordingly to generate a trained weight combination of each of the network layers. A quantization is performed on the trained weight combination of each of the network layers to generate a quantized neural network, in which each of the network layers thereof includes the trained weight combination.