Quantized Neural Network Data Path for Fractional Precision
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network models require significant computational resources due to the use of floating-point formats, leading to substantial memory and processing overhead, which is exacerbated by the challenge of maintaining fractional precision with small bit-widths.
Innovation Solution
Implementing selective normalization and rounded right-shift operations in the data path of neural networks to retain maximal fractional precision while minimizing bit-width, using fixed-point integer representations and piecewise-linear approximations of activation functions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If floating-point formats are used to store and process model parameters and data, then computational precision is maintained, but memory space and computational overhead increase significantly
Solution Approach 1:
The patent applies parameter changes by transitioning from floating-point format to fixed-point quantized format for representing model parameters and activation data. This changes the numerical representation parameters, allowing the same computational precision to be achieved with reduced bit-width (e.g., 16-bit or 8-bit integers instead of 32-bit floats), thereby reducing memory space requirements while maintaining adequate computational accuracy for neural network operations
2Measurement precision
If floating-point formats are used to store and process model parameters and data, then computational precision is maintained, but computational overhead increases significantly
Solution Approach 1:
The patent changes the computational parameters from floating-point arithmetic to fixed-point integer arithmetic. This parameter change enables the use of simpler, faster integer-based computational operations in place of complex floating-point operations, reducing computational overhead and improving processing speed while maintaining sufficient precision through carefully designed quantization schemes that preserve the essential information needed for accurate neural network computations
3Quantity of substance
If quantization is applied to reduce memory and computational requirements, then resource consumption decreases, but maintaining fractional precision becomes challenging with small bit-widths
Solution Approach 1:
The patent applies segmentation by dividing the fixed-point number representation into distinct integer and fractional bit portions. This segmentation allows the system to allocate specific bits for fractional precision while using the remaining bits for integer values, enabling accurate representation of quantized data even with small total bit-widths. The segmented structure preserves fractional information-carrying bits that are crucial for maintaining computational accuracy in quantized neural networks
4Productivity
If smaller bit-widths are used for intermediate values, then computational efficiency improves, but information-carrying capacity decreases
Solution Approach 1:
The patent changes the parameters of intermediate value representation by using optimized fixed-point formats with carefully selected bit allocations. This parameter change allows smaller bit-widths to be used while maintaining adequate information-carrying capacity through techniques such as asymmetric quantization ranges, dynamic scaling factors, and selective precision allocation different from conventional approaches, thereby improving computational efficiency without excessive information loss
Data Source
AI summary
Certain aspects of the present disclosure provide techniques and apparatus for quantized machine learning. A quantized input matrix is accessed at a layer of a neural network, and a first interim value is generated in an accumulator by performing matrix multiplication, using the accumulator, of the quantized input matrix and a quantized weight matrix associated with the layer of the neural network. The first interim value is normalized based at least in part on one or more leading sign bits of the first interim value, and the normalized first interim value is dequantized. A second interim value is generated by applying a rounded right-shift operation to the dequantized normalized first interim value, and activation data is generated by applying an activation function to the second interim value.