Neural Model Search With Layer-Wise Quantization for Mobile Deployment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The deployment of deep neural networks on mobile terminals is hindered by their limited computing capabilities and memory resources, and existing methods for mixed bit quantization and neural architecture search are inefficient due to frequent communication with the mobile terminal during the quantization and search process.
Innovation Solution
Generate multiple single bit models based on a to-be-quantized model, measure evaluation parameters on a mobile terminal, and perform model search by determining quantization weights to optimize layer structures, reducing the need for frequent information exchange and improving efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If mixed bit quantization and neural architecture search are performed with frequent communication with mobile terminal, then model quantization accuracy can be obtained, but time consumption increases and search efficiency decreases
Solution Approach 1:
The patent pre-collects inference time data for different layer structures during model construction, before actual model search begins. This preliminary data collection eliminates the need for repeated terminal communication during search, as the computing device can directly use pre-measured data to evaluate and compare different quantization schemes, thereby reducing time consumption while maintaining accuracy evaluation capability.
Solution Approach 2:
The patent creates a virtual evaluation environment where the computing device uses pre-collected inference time data to simulate and evaluate model performance. Instead of repeatedly communicating with the actual mobile terminal during search, the system uses copied measurement data from pre-tests to guide the model search process, significantly reducing communication overhead and time consumption.
2Quantity of substance
If mixed bit quantization is applied to reduce memory size and inference time, then resource requirements decrease, but the complexity of quantization scheme selection increases
Solution Approach 1:
The patent implements a feedback mechanism where the computing device uses pre-collected inference time data to evaluate different quantization schemes and provides guidance on bit allocation for each layer. This feedback loop allows the system to automatically select optimal quantization schemes based on measured performance data, reducing the complexity of manual scheme selection while achieving resource optimization.
Solution Approach 2:
The patent systematically varies quantization parameters (bit depths for different layers) based on pre-measured inference time characteristics of different layer structures. By changing quantization parameters according to empirically measured data rather than theoretical calculations, the system simplifies the selection process while achieving optimal balance between model size and performance.
3Reliability
If deep neural network with complex structure is deployed on mobile terminal, then model accuracy can be maintained, but computing capability and memory resources are exceeded
Solution Approach 1:
The patent applies different quantization bit depths to different layers of the neural network based on their specific characteristics and measured inference time requirements. Rather than uniformly quantizing all layers, the system identifies which layers can tolerate higher quantization (lower bits) and which require higher precision (more bits), thereby maintaining overall model accuracy while reducing total resource requirements.
Solution Approach 2:
The patent segments the neural network into different layers and applies customized quantization schemes to each segment based on pre-measured inference time data. This segmentation allows the system to optimize resource usage layer-by-layer, maintaining accuracy in critical layers while aggressively compressing less sensitive layers, thus deploying complex models on resource-constrained mobile devices.
Data Source
AI summary
This application relates to the field of artificial intelligence technologies, and discloses a machine learning model search method, a related apparatus, and a device. In the method, before model search and quantization, a plurality of single bit models are generated based on a to-be-quantized model, and evaluation parameters of layer structures in the plurality of single bit models are obtained. Further, after a candidate model selected from a candidate set is trained and tested, to obtain a target model, a quantization weight of each layer structure in the target model may be determined based on a network structure of the target model and evaluation parameters of all layer structures in the target model, a layer structure with a maximum quantization weight in the target model is quantized, and a model obtained through quantization is added to the candidate set.