Neural Network Architecture Search via Quantization-Aware Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current Neural Network Architecture Search (NAS) methods do not fully capture the effects of quantization on network performance, leading to distorted Pareto fronts and inefficient optimization of the quantization step, especially for specific problem domains and hardware constraints.
Innovation Solution
A computer-implemented method for optimized NAS that incorporates quantization-aware training and expert knowledge to select optimal quantization characteristics, thereby mitigating performance degradation due to quantization and generating a Pareto front representative of deployed models on target hardware.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If quantization is applied to reduce model size and improve deployment efficiency, then memory usage and deployment efficiency are improved, but network performance and accuracy deteriorate
Solution Approach 1:
The patent applies quantization-aware training as a preliminary action during the NAS process. By simulating quantization effects during training, the model learns to compensate for quantization-induced performance degradation before actual deployment, thus maintaining network performance while achieving model size reduction
Solution Approach 2:
The patent dynamically adjusts quantization parameters (such as bit-width and quantization schemes) during the NAS process based on performance feedback. This allows the system to find optimal quantization configurations that balance model size reduction with performance preservation for each specific architecture
2Productivity
If traditional NAS processes are used without quantization awareness, then the search process is simpler and faster, but the generated architectures show distorted Pareto fronts and do not accurately represent deployed model performance
Solution Approach 1:
The patent implements feedback loops where quantization performance metrics are measured and fed back into the NAS process. This feedback mechanism allows the system to iteratively refine architecture selections based on actual quantization performance, ensuring the Pareto front accurately reflects deployed model characteristics
Solution Approach 2:
The patent introduces quantization-aware training as an intermediary process between architecture generation and final deployment evaluation. This intermediary step bridges the gap between theoretical architecture performance and actual deployed performance, providing more accurate Pareto front representations
3Device complexity
If quantization characteristics are not optimized during NAS, then the search process is faster and less complex, but performance degradation due to quantization is not mitigated
Solution Approach 1:
The patent merges quantization optimization into the NAS process itself, combining architecture search with quantization parameter optimization. This integration allows simultaneous optimization of both structural and quantization parameters, achieving performance mitigation without proportionally increasing complexity
Data Source
AI summary
A method for performing an optimized Neural Network Architecture Search (NAS). The method includes: providing a technical problem definition, the technical problem to be solved by a to be searched set of candidate neuronal networks, a training dataset, a search space and preferably a seed neuronal network architecture; training, by the NAS-process, each one of the set of candidate neuronal networks based on the at least a part of the training dataset; adjusting and/or fine-tuning, by the NAS-process, a network architecture of each one of the set of trained candidate neuronal networks via Quantization-Aware Training; converting, by the NAS-process, the respectively adjusted and/or fine-tuned network architecture of each one of the set of candidate neuronal networks to a framework.