Neural Network Quantization for Embedded Resource Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional multilayer neural network models require significant processor resources and memory, making them inefficient for embedded devices due to high memory overhead and cumulative errors from fixed-point conversions, which hinder their operation on devices like smartphones.
Innovation Solution
The approach involves dividing the neural network into sub-structures with a quantization layer at the tail, transferring operation parameters to update quantization threshold parameters, and converting floating-point operations to fixed-point, thereby reducing resource usage and maintaining accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If fixed-point operation is used to convert floating-point parameters for embedded devices, then processor resources are reduced and operation efficiency is improved, but cumulative errors occur and system performance deteriorates
Solution Approach 1:
The patent applies preliminary action by pre-calculating and storing quantization threshold parameters during the model training phase. These threshold parameters are computed in advance and saved in the model structure, eliminating the need for real-time fixed-point conversion during inference. This allows the model to run efficiently on embedded devices with limited processor resources while maintaining floating-point precision through the use of pre-computed quantization thresholds.
2Measurement precision
If multilayer neural network model is operated on GPU-based workstation or server, then better data processing effect is obtained, but processor resources and memory overhead are significantly occupied
Solution Approach 1:
The patent applies segmentation by dividing the neural network model into multiple sub-structures, where each sub-structure contains a quantization layer with pre-computed threshold parameters. This segmentation allows the model to be processed in smaller, more manageable units that can be efficiently executed on embedded devices with limited resources, while maintaining the overall processing effectiveness through the distributed quantization thresholds.
3Reliability
If conventional multilayer neural network model is used, then modeling performance is achieved, but memory overhead is high and operation on embedded devices is hindered
Solution Approach 1:
The patent applies parameter changes by transforming the model parameters through quantization. During training, continuous floating-point parameters are converted into discrete quantization threshold parameters that are stored in the model. This parameter transformation reduces the memory footprint and enables the model to be deployed on embedded devices while preserving modeling performance through the use of these quantized parameters during inference.
Data Source
AI summary
A method and an apparatus for optimizing and applying a multilayer neural network model, and a storage medium are provided. The optimization method includes, dividing out at least one sub-structure from the multilayer neural network model to be optimized, wherein a tail layer of the divided sub-structure is a quantization layer, and transferring operation parameters in layers other than the quantization layer to the quantization layer for each of the divided sub-structures and updating quantization threshold parameters in the quantization layer based on the transferred operation parameters. When a multilayer neural network model optimized based on the optimization method is operated, the necessary processor resources can be reduced.