Neural Network Weight Initialization via Quantization Loss Awareness
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques for training machine-trained neural networks to use quantized weight values do not adequately account for the impact of quantization on network accuracy, leading to potential errors in output predictions.
Innovation Solution
A method is introduced to train machine-trained neural networks using a loss function that constrains weight values to quantized discrete sets, such as ternary values, while accounting for the loss in accuracy due to quantization. This method initializes weight values from floating-point values to quantized values, using a loss-aware penalty term and a Hessian matrix approximation to minimize computational burden.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If floating-point weight values are used in neural networks, then network accuracy is maintained, but memory usage and power consumption increase significantly
Solution Approach 1:
The patent changes the parameter of weight values from floating-point format to quantized discrete values (e.g., ternary values -1, 0, 1). This parameter transformation reduces memory requirements while the loss function with penalty terms compensates for accuracy degradation, achieving a balance between memory efficiency and network performance
Solution Approach 2:
The patent applies different quantization strategies to different parts of the network. By using layer-specific scaling factors and selective quantization, it maintains higher accuracy in critical layers while achieving greater compression in less critical layers, optimizing the local quality of weight representation throughout the network
2Reliability
If floating-point weight values are used in neural networks, then network accuracy is maintained, but power consumption increases due to data porting and multiplication operations
Solution Approach 1:
The patent transforms weight values to quantized discrete formats, enabling replacement of multiplication operations with simpler addition/subtraction operations. This parameter change directly reduces computational power consumption while the loss-aware training framework ensures accuracy is preserved through compensatory mechanisms
Solution Approach 2:
The patent extracts and eliminates the need for complex multiplication operations by using quantized weights. By taking out the multiplication step and replacing it with addition/subtraction of discrete values, it significantly reduces the energy consumption associated with computational operations
3Quantity of substance
If quantized weight values are used in neural networks, then memory usage and power consumption are reduced, but network accuracy deteriorates due to quantization error
Solution Approach 1:
The patent implements feedback through the loss function that monitors accuracy degradation caused by quantization. The penalty terms in the loss function provide feedback signals that guide the training process to adjust quantized weights and scaling factors, compensating for quantization errors and maintaining network accuracy
Solution Approach 2:
The patent performs preliminary action by pre-calculating and storing scaling factors for each layer before final quantization. This preliminary preparation allows the network to adapt to quantization effects in advance, reducing the impact of quantization errors on final accuracy
4Quantity of substance
If quantized weight values are used in neural networks, then memory usage is reduced, but computational precision deteriorates due to discrete value constraints
Solution Approach 1:
The patent changes the parameter of weight representation to quantized discrete values, which reduces memory usage. To compensate for precision loss, it introduces layer-specific scaling factors that effectively expand the dynamic range of quantized values, thereby maintaining computational precision despite the discrete nature of the weights
Data Source
AI summary
Some embodiments of the invention provide a method for configuring a network with multiple nodes. Each node generates an output value based on received input values and a set of weights that are previously trained to each have an initial value. For each weight, the method calculates a factor that represents a loss of accuracy to the network due to changing the weight from its initial value to a different value in a set of allowed values for the weight. Based on the factors, the method identifies a subset of the weights that have factors with values below a threshold. The method changes the values of each weight from its initial value to one of the values in its set of allowed values. The values of the identified subset are all changed to zero. The method trains the weights beginning with the changed values for each weight.