Conditional Predictor Finetuning for Hardware-Aware Neural Architecture Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Balancing accuracy and computation cost is a challenge in training learning-based predictors for neural architecture search (NAS), as purely offline training lacks accuracy for unseen architectures, while purely online training is slow and resource-intensive.
Innovation Solution
A conditional finetuning mechanism that combines offline and online training, where the predictor is initially trained offline and finetuned only when predicted performance significantly deviates from validated performance, using the reference DNN and target setting to improve accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If the predictor is trained purely offline with general datasets, then computation cost is reduced, but prediction accuracy for unseen architectures and target settings deteriorates
Solution Approach 1:
The system performs preliminary offline training with general datasets to establish a baseline predictor, then applies conditional finetuning only when accuracy thresholds are not met. This preliminary action reduces computation cost by avoiding full online training while maintaining the capability to improve accuracy when needed.
Solution Approach 2:
The system dynamically changes training parameters by switching between offline and online training modes based on accuracy requirements. The conditional finetuning mechanism adjusts the training regime (from general offline training to targeted online finetuning) based on the detected performance gap, optimizing the balance between computation cost and prediction accuracy.
2Measurement precision
If the predictor is trained purely online with target-specific data, then prediction accuracy improves, but validation time and resource consumption increase
Solution Approach 1:
Instead of applying uniform online training across all cases, the system applies conditional finetuning locally only to the extent necessary. The finetuning is triggered only when the prediction accuracy gap exceeds a threshold, and only for the specific target setting and architecture type that exhibits the gap, thereby reducing overall validation time while maintaining accuracy where needed.
Solution Approach 2:
The system uses partial online training (conditional finetuning) rather than complete online retraining. By applying finetuning only when and where accuracy thresholds are not met, the system achieves sufficient accuracy improvement without the full time cost of comprehensive online training.
3Measurement precision
If conditional finetuning is applied, then prediction accuracy for target settings improves, but device complexity increases
Solution Approach 1:
The conditional finetuning mechanism is self-regulating, automatically determining when finetuning is needed by comparing prediction accuracy against thresholds. The system monitors its own performance and initiates finetuning only when necessary, reducing the need for external control complexity while maintaining high accuracy.
Solution Approach 2:
The system implements a feedback loop where prediction accuracy is continuously monitored and compared against thresholds. This feedback mechanism automatically triggers conditional finetuning when accuracy gaps are detected, creating a self-correcting system that improves accuracy without requiring complex manual intervention or control mechanisms.
Data Source
AI summary
A predictor of sub-network performance is conditionally finetuned. The predictor is a neural network that has been trained offline with a general set of neural networks and a general set of hardware and software configurations. The validated performance of a set of sub-networks on a hardware platform is calculated, where the hardware platform is configured with a target setting. Each sub-network is a sub-model of a super-network generated based on a reference deep learning neural network (DNN). Then the predictor calculates predicted performance of the set of sub-networks. The predictor is finetuned in response to a pre-search determination that a difference between the predicted performance and the validated performance exceeds a threshold. The finetuning is based on, at least in part, the reference DNN and the target setting. A network architecture search (NAS) performs in multiple iterations using a same predictor in each iteration and outputs a final network model.