Neural Network Quantization via Layer Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network devices face challenges in processing complex input data in real-time due to high computational requirements, particularly in low-powered embedded systems like smartphones, where minimizing accuracy loss while reducing computation is necessary.
Innovation Solution
A method for neural network quantization that involves performing forward and backward passes on a first neural network with a specific bit precision, obtaining profile information on gradients and weights, determining layers to be quantized with lower precision, and generating a second neural network by quantizing those layers without retraining, while maintaining accuracy within a predetermined threshold.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If neural network devices use high bit precision to maintain accuracy, then manufacturing precision is improved, but use of energy and computational load increase
Solution Approach 1:
The patent divides the neural network into different segments or layers, applying different bit precisions to different layers based on their importance and sensitivity to quantization. This allows critical layers to maintain high precision while less critical layers use lower precision, reducing overall computational load and energy consumption while preserving accuracy where it matters most.
Solution Approach 2:
The patent implements local quality by assigning different bit precisions to different parts (layers) of the neural network based on their specific characteristics and importance. Instead of uniform quantization, each layer is analyzed and quantized according to its own requirements, allowing high accuracy in sensitive layers while achieving energy efficiency in less sensitive layers.
2Manufacturing precision
If neural network devices use high bit precision, then manufacturing precision is improved, but device complexity increases
Solution Approach 1:
The patent segments the neural network computation into different precision levels, organizing layers into groups that require different bit precisions. This segmentation simplifies the overall device complexity by allowing specialized handling of different precision requirements rather than uniformly high-precision computation throughout the entire network.
Solution Approach 2:
By applying local quality, the patent optimizes device complexity by matching computational precision to the actual needs of each layer. This avoids the unnecessary complexity of high-precision arithmetic in layers where lower precision suffices, while maintaining required accuracy in critical layers.
3Use of energy by moving object
If quantization is applied to reduce computational load, then use of energy is reduced, but manufacturing precision deteriorates
Solution Approach 1:
The patent applies segmentation by dividing the neural network into quantized and non-quantized portions, or different quantization levels, based on layer importance. This allows energy-efficient quantization in appropriate layers while preserving accuracy in layers where quantization would cause unacceptable degradation.
Solution Approach 2:
Through local quality, the patent enables energy-efficient quantization in specific layers where it has minimal impact on accuracy, while maintaining high precision in layers where accuracy is critical. This selective approach optimizes the trade-off between energy efficiency and manufacturing precision on a layer-by-layer basis.
4Productivity
If quantization is applied to reduce computational load, then productivity is improved, but manufacturing precision deteriorates
Solution Approach 1:
The patent improves productivity by segmenting the neural network into layers that can be processed with lower precision, enabling faster computation in those segments. By identifying and quantizing appropriate layers, the system achieves accelerated inference speed while maintaining accuracy in critical unquantized layers.
Solution Approach 2:
Through local quality, the patent optimizes inference speed by applying quantization selectively to layers where it provides sufficient accuracy, thereby achieving speed improvements without unacceptable accuracy loss. Critical layers maintain high precision to preserve overall model accuracy while non-critical layers benefit from faster quantized computation.
Data Source
AI summary
A method for neural network quantization includes performing a forward pass and a backward pass of a first neural network having a first bit precision with respect to each of a plurality of input data sets, obtaining profile information with respect to at least one of input gradients, weight gradients, and output gradients calculated for each layer of layers included in the first neural network in the process of performing the backward pass, determining one or more layers, from among the layers, to be quantized with a second bit precision less than the first bit precision, based on the obtained profile information, and generating a second neural network by quantizing the determined layers from among the layers with the second bit precision.