Per-Channel Fixed-Point Neural Quantization for Low-Power Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks face challenges in processing large amounts of data in real-time due to the complexity of floating-point operations, which can lead to inefficiencies and accuracy losses, especially in low-processing-power devices like mobile devices.
Innovation Solution
A method is introduced to generate a fixed-point quantized neural network by analyzing the statistical distribution of floating-point parameter values, determining fractional lengths for biases and weights, and bit-shifting to optimize fixed-point expressions, allowing for more efficient operations while maintaining accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If floating-point operations are used for neural network processing, then calculation precision is maintained, but processing speed and efficiency deteriorate on low-processing-power devices
Solution Approach 1:
The patent changes the numerical representation parameter from floating-point to fixed-point format. By analyzing the statistical distribution of parameter values and determining appropriate fractional lengths for biases and weights, the system converts neural network parameters to fixed-point expressions, enabling efficient processing on low-power devices while maintaining acceptable accuracy through post-quantization fine-tuning
Solution Approach 2:
The patent performs preliminary statistical analysis of the neural network parameters before quantization. By obtaining statistics for each channel of floating-point parameter values and analyzing their distribution ranges, the system determines optimal fixed-point expressions in advance, which guides the subsequent quantization process and ensures precision is preserved where needed
2Measurement precision
If floating-point parameters are used in neural network, then accuracy is maintained, but device complexity and computational overhead increase
Solution Approach 1:
The patent changes the numerical representation parameter from floating-point to fixed-point format. By analyzing the statistical distribution of parameter values and determining appropriate fractional lengths for biases and weights, the system converts neural network parameters to fixed-point expressions, enabling efficient processing on low-power devices while maintaining acceptable accuracy through post-quantization fine-tuning
Solution Approach 2:
The patent segments the neural network parameters into different groups (biases and weights) and determines fractional lengths for each group separately based on their statistical distributions. This segmentation allows targeted optimization where different parameter types can use different precision levels, reducing overall computational overhead while maintaining accuracy for critical parameters
3Productivity
If fixed-point quantization is applied to neural network parameters, then processing efficiency is improved, but measurement precision may deteriorate
Solution Approach 1:
The patent changes the numerical representation parameter from floating-point to fixed-point format. By analyzing the statistical distribution of parameter values and determining appropriate fractional lengths for biases and weights, the system converts neural network parameters to fixed-point expressions, enabling efficient processing on low-power devices while maintaining acceptable accuracy through post-quantization fine-tuning
Solution Approach 2:
The patent implements a feedback mechanism through post-quantization fine-tuning. After converting parameters to fixed-point format, the system trains the quantized neural network again on the training dataset, adjusting the fixed-point parameters to compensate for quantization errors. This feedback loop restores accuracy that may have been lost during the quantization process
Data Source
AI summary
A method of generating a fixed-point quantized neural network includes analyzing a statistical distribution for each channel of floating-point parameter values of feature maps and a kernel for each channel from data of a pre-trained floating-point neural network, determining a fixed-point expression of each of the parameters for each channel statistically covering a distribution range of the floating-point parameter values based on the statistical distribution for each channel, determining fractional lengths of a bias and a weight for each channel among the parameters of the fixed-point expression for each channel based on a result of performing a convolution operation, and generating a fixed-point quantized neural network in which the bias and the weight for each channel have the determined fractional lengths.