AI Model Quantization via Structure Search for Edge Deployment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The deployment of complex AI models on terminal devices with limited computing resources is hindered by high hardware requirements and the need for manual, model-specific quantization, which affects quantization speed and generality.
Innovation Solution
A general quantization method that involves structure search and the use of fake operators, including quantization and dequantization operators, to automatically optimize AI models for low-precision processing, reducing labor costs and improving quantization speed while maintaining generality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual quantization methods are used to deploy AI models on terminal devices, then model precision can be maintained, but code invasiveness increases and quantization speed decreases
Solution Approach 1:
The system performs self-service by automatically searching for quantization schemes without manual intervention. The structure search mechanism autonomously explores different quantization configurations and selects optimal schemes, eliminating the need for manual code modification while maintaining model precision.
Solution Approach 2:
A structure search mechanism acts as an intermediary between the AI model and quantization process. This mediator automatically analyzes model structures, identifies suitable quantization schemes, and applies them without direct manual intervention, thereby improving quantization speed while preserving precision.
2Measurement precision
If manual quantization is performed for different model structures, then model precision can be maintained, but code reusability decreases and labor costs increase
Solution Approach 1:
The structure search mechanism provides universal quantization capability that works across different AI model structures. By automatically adapting to various model types without requiring manual intervention for each specific structure, the system achieves both precision maintenance and quantization generality.
Solution Approach 2:
The system changes parameters automatically through structure search rather than manual configuration. The quantization scheme parameters are dynamically adjusted based on model structure analysis, enabling the same quantization framework to work universally across different models while maintaining optimal precision for each.
3Reliability
If AI models with larger weight parameters are used, then model performance improves, but hardware requirements and energy consumption increase
Solution Approach 1:
The system changes the precision parameters of model weights through automatic quantization. By converting high-precision floating-point parameters to lower-precision representations, the system reduces energy consumption and hardware requirements while maintaining model performance through structure-search-based optimization.
Solution Approach 2:
The patent replaces complex high-precision computational mechanics with simpler low-precision operations. Through automatic quantization, the system substitutes resource-intensive floating-point calculations with more efficient integer or reduced-precision operations, reducing energy consumption while preserving model performance.
Data Source
AI summary
A data processing method includes: performing, when obtaining an initial model, structure search on the initial model to obtain a model structure diagram of the initial model; quantizing, based on a type of a target structure in the model structure diagram of the initial model, the target structure of the initial model by applying a fake operator for quantization, the fake operator comprising a quantization operator and a dequantization operator; and obtaining, based on the quantized initial model, a service model for service processing.