Hardware-Aware Joint Model Architecture And Quantization Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional neural network (NN) workflows for embedded applications optimize model architecture and quantization in disconnected stages, failing to consider memory limitations and resulting in suboptimal performance on resource-constrained devices.
Innovation Solution
The MCMP-NAS system jointly optimizes machine learning model architecture and quantization precision for each layer under memory constraints, using a Once-For-All (OFA) hardware-aware NAS framework to identify the optimal architecture and quantization levels for resource-constrained hardware.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If model architecture and quantization are optimized in disconnected stages, then the design process is simpler, but the performance on resource-constrained devices is suboptimal
Solution Approach 1:
The patent merges architecture search and quantization optimization into a unified joint optimization framework. The system simultaneously searches for optimal model architectures and quantization configurations across multiple bit-widths (4-bit, 8-bit, 16-bit) within a single search space, eliminating the need for separate disconnected optimization stages and enabling end-to-end hardware-aware optimization.
Solution Approach 2:
The system dynamically changes quantization parameters (bit-widths) for different layers and architectures during the joint optimization process. By treating quantization precision as a searchable parameter rather than a fixed constraint, the system can adaptively select optimal precision levels for each layer based on hardware constraints and performance requirements.
2Device complexity
If fixed bit width quantization is used, then the implementation is simpler, but the model size and performance on resource-constrained hardware are suboptimal
Solution Approach 1:
The patent implements mixed-precision quantization where different layers of the neural network use different bit-widths (4-bit, 8-bit, or 16-bit) based on their specific requirements. This local differentiation allows critical layers to maintain higher precision while less critical layers use lower precision, optimizing the trade-off between model size and performance without requiring uniform quantization across the entire model.
Solution Approach 2:
The system transitions from static fixed-bit-width quantization to dynamic mixed-precision quantization. The joint optimization framework dynamically determines the optimal bit-width assignment for each layer based on hardware constraints and performance targets, enabling adaptive precision selection rather than committing to a single fixed precision level for the entire model.
3Reliability
If higher precision quantization is used, then the model performance is better, but the memory requirements and latency on resource-constrained devices increase
Solution Approach 1:
The system changes the precision parameter (bit-width) dynamically based on hardware constraints and performance requirements. By integrating latency and memory constraints directly into the joint optimization objective, the system automatically selects appropriate precision levels for each layer to meet timing requirements while maintaining acceptable performance, rather than using uniformly high precision that would exceed latency budgets.
Solution Approach 2:
Different layers are assigned different precision levels (4-bit, 8-bit, or 16-bit) based on their individual impact on performance and computational cost. This localized precision assignment allows the system to maintain high precision only where necessary for performance-critical operations while using lower precision for other layers, thereby reducing overall inference latency and memory usage while preserving model accuracy.
Data Source
AI summary
Methods and apparatus are disclosed for joint optimization of machine learning model architecture and quantization. An example method includes generating a first machine learning model for a resource-constrained device based on quantized outputs from each of a plurality of compute blocks. Each compute block includes a plurality of inverted residual blocks coupled in series. Determining the quantized output of each respective compute block includes performing a first convolution, based at least in part on a first quantization level, on input data to a first inverted residual block, performing a second convolution on an output of the first convolution based at least in part on the first quantization level, adding an output of the second convolution to the input data to generate a first quantized output, and providing the first quantized output to a second inverted residual block, and providing the first machine learning model to the resource-constrained device for execution.