Learned Step Size Quantization for Neural Network Precision
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks face challenges in maintaining high accuracy when reducing precision, particularly at extremely low precision levels, due to the difficulty in optimally configuring the quantizer's step size for weight and activation layers.
Innovation Solution
The implementation of Learned Step Size Quantization (LSQ), which learns the quantization mapping for each layer, approximates the gradient of the quantizer step size sensitive to quantized state transitions, and balances step size updates with weight updates, allowing for finer grained optimization and improved convergence.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If quantization precision is reduced to improve computational efficiency and reduce model size, then productivity and energy efficiency are improved, but measurement precision and manufacturing precision deteriorate
Solution Approach 1:
The patent applies dynamics by making the quantizer step size configurable and learnable rather than fixed. The step size is optimized during training to adapt to different layers and data distributions, allowing the system to dynamically adjust precision levels to maintain accuracy while enabling lower precision operations where appropriate.
Solution Approach 2:
The patent changes the parameter of quantizer step size from a fixed value to a learnable parameter that is optimized during training. This allows the system to find optimal step sizes that balance precision and efficiency for each specific layer and application, resolving the contradiction between reduced precision and maintained accuracy.
2Device complexity
If quantizer step size is fixed to simplify configuration, then device complexity is reduced, but adaptability deteriorates
Solution Approach 1:
The patent applies self-service by making the quantizer step size learnable during training. The system automatically optimizes its own configuration through gradient-based optimization, eliminating the need for manual configuration while achieving layer-specific adaptation. The quantizer serves itself by learning optimal parameters from the data.
3Measurement precision
If gradient approximation through quantizer is improved to enable better optimization, then measurement precision is improved, but computational complexity increases
Solution Approach 1:
The patent introduces an intermediary approach by using a configurable step size that acts as a bridge between the quantized and continuous domains. This intermediary parameter enables gradient flow through the quantizer during training, allowing for more accurate optimization while maintaining the benefits of quantization.
Data Source
AI summary
Learned step size quantization in artificial neural network is provided. In various embodiments, a system comprises an artificial neural network and a computing node. The artificial neural network comprises: a quantizer having a configurable step size, the quantizer adapted to receive a plurality of input values and quantize the plurality of input values according to the configurable step size to produce a plurality of quantized input values, at least one matrix multiplier configured to receive the plurality of quantized input values from the quantizer and to apply a plurality of weights to the quantized input values to determine a plurality of output values having a first precision, and a multiplier configured to scale the output values to a second precision. The computing node is operatively coupled to the artificial neural network and is configured to: provide training input data to the artificial neural network, and optimize the configurable step size based on a gradient through the quantizer and the training input data.