Multi-Stage ML Model Synthesis for Edge Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models for edge devices face challenges in achieving high accuracy and low latency due to the isolation of perspectives in design, such as novel network architectures not being well-supported by hardware accelerators, which limits their practical improvements and broader applicability.
Innovation Solution
A multi-stage process for synthesizing a combined model that uses a lightweight prediction model to generate combination values for a set of basis models, allowing for early termination and efficient inference, while maintaining accuracy, by processing inputs through a CPU and specialized accelerators, and enabling flexible hardware execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If novel network architectures are used to improve accuracy, then model accuracy is improved, but hardware support becomes insufficient and device performance deteriorates
Solution Approach 1:
The model is segmented into a base model and multiple specialized basis models. The base model handles general cases efficiently with hardware-friendly architecture, while basis models specialize in specific scenarios. This segmentation allows the system to achieve high accuracy through selective model composition without requiring the base architecture to support all specialized operations natively.
Solution Approach 2:
The base model is designed as a universal architecture that can serve multiple functions: it acts as a standalone model for efficient inference and as a foundation for synthesizing specialized models through learned combinations. This multi-functionality allows a single hardware-optimized architecture to achieve both speed and accuracy by dynamically composing different basis models based on input characteristics.
2Measurement precision
If model complexity is increased to improve accuracy, then accuracy is improved, but computational cost and latency increase
Solution Approach 1:
The system dynamically synthesizes model parameters at inference time by learning combination coefficients that adapt to input characteristics. Instead of using a single static complex model, the system dynamically composes lightweight basis models with learned weights, achieving high accuracy only when needed while maintaining efficiency for simpler cases through the base model.
Solution Approach 2:
The approach changes model parameters dynamically through learned combination coefficients rather than using a fixed complex architecture. The base model parameters remain simple and hardware-friendly, while the effective model complexity is adjusted through linear combinations of basis models with learned weights, achieving accuracy improvements without proportional increases in computational cost.
3Measurement precision
If specialized models are created for different scenarios to improve accuracy, then accuracy is improved, but device complexity and deployment difficulty increase
Solution Approach 1:
Multiple specialized basis models are merged into a unified framework where they are synthesized through learned linear combinations. Instead of maintaining separate deployment paths for different models, the system merges them into a single cohesive architecture where the base model and basis models work together through parameter synthesis, simplifying deployment while maintaining scenario-specific accuracy.
Data Source
AI summary
Example implementations of the present disclosure combine efficient model design and dynamic inference. With a standalone lightweight model, the unnecessary computation on easy examples is avoided and the information extracted by the lightweight model also guide the synthesis of a specialist network from the basis models. With extensive experiments on ImageNet it is shown that a proposed example BasisNet is particularly effective for image classification and a BasisNet-MV3 achieves 80.3% top-1 accuracy with 290 M MAdds without early termination.