Neural Network Weight Quantization via Anchor Point Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks require significant memory resources, making them inefficient for deployment in resource-constrained environments, such as mobile devices, due to their large memory footprint and lack of flexibility in adapting to varying memory constraints.
Innovation Solution
A method for neural network compression that involves training the network using a loss function that encourages weight values to approach anchor points, followed by quantization, reducing the number of unique weight values and thus the memory footprint, while maintaining prediction accuracy. This method allows for the generation of multiple compressed networks with varying memory footprints and prediction accuracies, enabling efficient deployment across different environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If neural network weight values are stored with high precision, then prediction accuracy is maintained, but memory footprint increases significantly
Solution Approach 1:
The patent changes the parameter of weight value precision from high-precision floating-point representation to low-precision quantized representation. By training the network to learn quantized weight values directly (using anchor points and quantization loss), the system achieves comparable prediction accuracy while dramatically reducing memory footprint, as quantized weights require far fewer bits to store.
2Quantity of substance
If neural network is compressed to reduce memory footprint, then deployment in resource-constrained environments becomes feasible, but prediction accuracy may deteriorate
Solution Approach 1:
The patent applies preliminary quantization during the training phase rather than as a post-processing step. By integrating quantization loss into the training objective and using anchor points to guide weight quantization beforehand, the network learns to accommodate quantized representations, thereby maintaining prediction accuracy even after compression to reduced memory footprint.
3Use of energy by moving object
If quantization is applied to reduce memory usage, then resource efficiency improves, but network flexibility and adaptability decrease
Solution Approach 1:
The patent introduces dynamic anchor points that can be updated during training iterations. These anchor points adapt to the distribution of weight values as training progresses, allowing the quantization scheme to dynamically adjust to the network's learning needs. This dynamic adaptation maintains network flexibility while achieving efficient resource utilization through quantization.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for neural network compression. In one aspect, a method comprises receiving a neural network and identifying a particular set of multiple weights of the neural network. Multiple anchor points are determined based on current values of the particular set of weights of the neural network. The neural network is trained by, at each of multiple training iterations, performing operations comprising adjusting the values of the particular set of weights by backpropagating gradients of a loss function. The loss function comprises a first loss function term based on a prediction accuracy of the neural network and a second loss function term based on a similarity of the current values of the particular set of weights to the anchor points. After training, the values of the particular set of weights are quantized based on the anchor points.