Neural Network Quantized Weight Training Method
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning techniques for training neural networks face challenges in reducing memory and power consumption, particularly due to large floating-point weight values, which require significant memory and power for storage and computation, and quantization methods often introduce accuracy errors without considering the impact on network accuracy.
Innovation Solution
A method is introduced to configure machine-trained neural networks with quantized weight values using a loss function that constrains weights to discrete sets, such as ternary values, while accounting for accuracy loss, using a loss-aware penalty term and empirical Fisher approximation to minimize computational burden and optimize weight initialization and sparsity conditions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If floating-point weight values are used in neural networks, then computational accuracy is maintained, but memory consumption and power usage increase significantly
Solution Approach 1:
The patent applies parameter changes by transitioning weight values from floating-point format to quantized discrete values (e.g., binary or ternary). This fundamental parameter change reduces the precision requirement while enabling more efficient storage and computation, thereby reducing power consumption and memory usage without completely sacrificing accuracy
Solution Approach 2:
The patent applies local quality by introducing layer-specific scaling factors that allow different layers of the neural network to have different quantization characteristics. This enables critical layers to maintain higher effective precision while less critical layers use coarser quantization, optimizing the balance between accuracy and efficiency locally across different network components
2Measurement precision
If floating-point weight values are used in neural networks, then computational accuracy is maintained, but memory storage requirements increase
Solution Approach 1:
The patent changes the parameter representation from floating-point to quantized discrete values, dramatically reducing the number of bits required to store each weight. This parameter transformation enables storage of large neural networks in limited on-chip memory, eliminating the need for frequent off-chip memory access
Solution Approach 2:
The patent segments the weight storage by layer, with each layer having its own scaling factor and quantization characteristics. This segmentation allows efficient packing of weight values in memory while maintaining the ability to reconstruct accurate values during computation through the use of layer-specific scaling factors
3Use of energy by moving object
If weight values are quantized to discrete values, then memory and power consumption are reduced, but accuracy errors are introduced
Solution Approach 1:
The patent introduces scaling factors as additional parameters that compensate for the precision loss from quantization. By multiplying quantized weight values by layer-specific scaling factors during computation, the network can recover much of the accuracy that would otherwise be lost due to discrete quantization, thus reducing the accuracy-error penalty while maintaining low power consumption
Solution Approach 2:
The patent creates a composite representation by combining quantized discrete weight values with continuous scaling factors. This composite approach allows the system to benefit from the efficiency of discrete values for storage and basic computation while using the continuous scaling factors to correct and refine the results, achieving a balance between efficiency and accuracy
4Ease of manufacture
If uniform quantization is applied across all layers, then implementation simplicity is maintained, but network accuracy is compromised
Solution Approach 1:
The patent applies local quality by assigning different scaling factors to different layers of the neural network. This allows each layer to be optimized independently, with critical layers receiving higher precision (larger scaling factors) and less critical layers using coarser quantization. This local differentiation maintains or improves accuracy compared to uniform quantization while still being simpler than fully custom per-weight precision
Data Source
AI summary
Some embodiments of the invention provide a method for configuring a machine-trained (MT) network that includes multiple nodes. Each node of a set of the nodes generates an output value based on received input values and a set of configurable weights. The method propagates a set of inputs through the MT network to generate a set of outputs, with each input having a corresponding expected output. The method calculates a value of a loss function comprising (i) a first term that measures a difference between each generated output and its corresponding expected output and (ii) a second term that constrains the weights to discrete sets of allowed values and accounts for an increase in the first term due to constraining the weights to the discrete sets of values. The method uses the calculated value of the loss function to train the weights of the MT network.