Neural Network Weight Initialization Using Quantized Values
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning techniques for training neural networks face challenges in reducing memory and power consumption, particularly due to large floating-point weight values, which require significant memory and power for storage and computation, and quantization methods often introduce accuracy errors without considering the impact on network accuracy.
Innovation Solution
A method is introduced to configure machine-trained neural networks with quantized weight values using a loss function that constrains weights to discrete sets, such as ternary values, while accounting for accuracy loss, using a loss-aware penalty term and empirical Fisher approximation to minimize computational burden and optimize weight initialization and sparsity conditions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If floating-point weight values are used in neural networks, then the network can achieve high accuracy in predictions, but the memory consumption and power consumption increase significantly
Solution Approach 1:
The patent changes the parameter of weight representation from floating-point to quantized discrete values. This parameter change reduces the memory footprint and computational complexity while the loss function and penalty terms ensure that the quantized network maintains prediction accuracy comparable to floating-point networks
Solution Approach 2:
The patent applies different quantization strategies to different parts of the network. Specifically, different scaling factors are used for different layers, and the sparsity condition is applied selectively. This local differentiation allows the network to maintain accuracy in critical regions while achieving compression in less sensitive areas
2Measurement precision
If floating-point weight values are used in neural networks, then the network can achieve high accuracy in predictions, but the memory storage requirements increase significantly
Solution Approach 1:
The patent changes the parameter of weight representation from floating-point to quantized discrete values. This parameter change reduces the memory footprint and computational complexity while the loss function and penalty terms ensure that the quantized network maintains prediction accuracy comparable to floating-point networks
Solution Approach 2:
The patent extracts only the essential information needed for accurate predictions by quantizing the weight values to discrete levels. This extraction process removes redundant precision while retaining the critical features necessary for network performance
3Quantity of substance
If quantization is applied to reduce memory consumption, then memory usage decreases, but the accuracy of the neural network output deteriorates
Solution Approach 1:
The patent incorporates feedback mechanisms through the loss function that monitors prediction accuracy during training. The loss function includes penalty terms that provide feedback to adjust the quantized weights, ensuring that accuracy is maintained despite the quantization process
Solution Approach 2:
The patent performs preliminary training with floating-point weights to establish optimal weight values before quantization. This preliminary action allows the network to learn the correct decision boundaries, which are then preserved during the quantization process through careful initialization and penalty terms
4Quantity of substance
If previous quantization techniques are used, then memory consumption is reduced, but the impact on network accuracy is not accounted for
Solution Approach 1:
The patent incorporates feedback mechanisms through the loss function that monitors prediction accuracy during training. The loss function includes penalty terms that provide feedback to adjust the quantized weights, ensuring that accuracy is maintained despite the quantization process
Solution Approach 2:
The patent makes the quantization process dynamic by allowing different scaling factors for different layers and by adjusting the sparsity condition during training. This dynamic approach allows the network to adapt to the accuracy requirements of different regions while maintaining overall performance
Data Source
AI summary
Some embodiments of the invention provide a method for configuring a network with multiple nodes. Each node generates an output value based on received input values and a set of weights that are previously trained to each have an initial value. For each weight, the method calculates a factor that represents a loss of accuracy to the network due to changing the weight from its initial value to a different value in a set of allowed values for the weight. Based on the factors, the method identifies a subset of the weights that have factors with values below a threshold. The method changes the values of each weight from its initial value to one of the values in its set of allowed values. The values of the identified subset are all changed to zero. The method trains the weights beginning with the changed values for each weight.