Hardware-Aware Neural Network Design Using Blockwise Knowledge Distillation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Hand-optimizing efficient neural networks for visual recognition on mobile and automotive hardware platforms is resource-intensive and time-consuming, requiring significant engineering effort and repeated processes due to inefficiencies in existing neural architecture search methods.
Innovation Solution
The method involves using an accuracy predictor built with blockwise knowledge distillation to select and fine-tune neural networks, reducing the need for manual optimization by automating the design process through evolutionary search and knowledge distillation, allowing for faster adaptation to changing hardware and software configurations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If hand-optimizing neural networks is performed, then efficiency and accuracy can be tuned, but the process is prohibitively time and resource consuming
Solution Approach 1:
The patent performs preliminary action by pre-training a teacher network and extracting its knowledge through knowledge distillation before the actual design process. The accuracy predictor is built in advance using blockwise knowledge distillation trained search blocks, so that when a new neural network needs to be designed, the predictor can immediately provide accuracy estimates without requiring time-consuming retraining, thus resolving the contradiction between optimization quality and design time
Solution Approach 2:
The patent creates a copy of the complex neural network design process through the accuracy predictor model. Instead of manually optimizing networks from scratch, the system uses the pre-built predictor to simulate and evaluate network performance, effectively copying the evaluation function without requiring the full computational cost of actual training, thereby reducing time loss while maintaining optimization reliability
2Reliability
If neural networks are trained from scratch, then accuracy can be achieved, but it requires around 400-500 GPU-hours each
Solution Approach 1:
The patent extracts the essential knowledge from the complex training process through knowledge distillation. By training search blocks using blockwise knowledge distillation and building an accuracy predictor from these blocks, the system extracts the critical accuracy information without requiring the full 400-500 GPU-hours of training. This extracted knowledge can then be used to evaluate and select networks much faster and with fewer resources
Solution Approach 2:
The accuracy predictor serves as an intermediary between the complex training process and the network selection process. Instead of directly training networks from scratch (which consumes 400-500 GPU-hours), the system uses the predictor as a mediator to estimate accuracy, significantly reducing the computational resources required while still achieving reliable accuracy assessment for network selection
3Adaptability or versatility
If the design process is repeated for quality level changes or hardware metric changes, then adaptability is improved, but the process must be repeated from scratch
Solution Approach 1:
The patent applies preliminary action by pre-building the accuracy predictor with blockwise knowledge distillation trained search blocks before any specific hardware or quality requirements are encountered. This pre-built predictor can immediately adapt to different hardware configurations and quality targets without requiring retraining, as the predictor already contains the necessary evaluation logic and knowledge representations, thus enabling rapid adaptation while avoiding repeated time-consuming design processes
Data Source
AI summary
Various aspects provide methods for a computing device selecting a neural network for a hardware configuration including using an accuracy predictor to select from a search space a neural network including a first plurality of the blockwise knowledge distillation trained search blocks, in which the accuracy predictor is built using search space trained blockwise knowledge distillation search blocks. Aspects may include selecting a second plurality of the blockwise knowledge distillation trained search blocks based on criteria of predicted accuracy using the accuracy predictor for the second plurality of the blockwise knowledge distillation trained search blocks. Aspects may include selecting the neural network based on a search of the blockwise knowledge distillation trained search blocks, initializing the blockwise knowledge distillation trained search blocks of the neural network using weights of the blockwise knowledge distillation trained search blocks, and fine-tuning the neural network using knowledge distillation, to generate a distilled neural network.