Deep Learning Accelerator Performance Prediction Using Instruction Features
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for refining deep learning models for edge devices are laborious and inefficient, with cycle-approximate simulators being time-consuming, and existing predictors lack accuracy in predicting inference latency due to insufficient hardware-informative features.
Innovation Solution
An instruction-based predictor is trained using machine instructions and intermediate representations to accurately predict inference latency, power consumption, and memory usage by simulating execution on edge devices, leveraging hardware-informative features.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If cycle-approximate simulators are used to predict performance metrics, then comprehensive performance data can be obtained, but the process becomes time-consuming and reduces productivity
Solution Approach 1:
The patent pre-computes and stores hardware-informative features (execution cycle counts, power consumption, memory usage) for each instruction in the instruction set architecture during system initialization. This preliminary action eliminates the need for time-consuming cycle-approximate simulation during the model refinement process, as the predictor can directly query pre-computed values to estimate performance metrics of neural network models.
2Measurement precision
If manual refinement of deep learning models is performed, then model performance can be optimized, but the process is laborious and time-consuming
Solution Approach 1:
The patent implements an automated predictor system that autonomously estimates performance metrics for different model configurations without requiring manual intervention. The system automatically compiles neural network models to the target instruction set, queries pre-computed hardware-informative features, and generates performance predictions, thereby eliminating the laborious manual refinement process while maintaining optimization quality.
Solution Approach 2:
The patent replaces the manual mechanical refinement process with an automated software-based predictor system. Instead of manually adjusting model parameters and performing time-consuming simulations, the system uses automated model compilation, feature extraction, and machine learning-based prediction to rapidly estimate performance metrics, substituting human labor with an efficient computational mechanism.
3Productivity
If existing predictors are used to estimate performance metrics, then refinement time can be reduced, but prediction accuracy is insufficient due to lack of hardware-informative features
Solution Approach 1:
The patent fundamentally changes the feature parameters used by predictors from high-level abstract representations to low-level hardware-informative features. By extracting execution cycle counts, power consumption, and memory usage directly from the instruction set architecture, the system transforms the predictor's input parameters to match actual hardware behavior more closely, thereby significantly improving prediction accuracy while maintaining refinement efficiency.
Data Source
AI summary
Aspects of the present disclosure provide a method for training a predictor that predicts performance of a plurality of machine learning (ML) models on platforms. For example, the method can include converting each of the ML models into a plurality of instructions or the instructions and a plurality of intermediate representations (IRs). The method can also include simulating execution of the instructions corresponding to each of the ML models on a platform and generating instruction performance reports. Each of the instruction performance reports can be associated with performance of the instructions corresponding to one of the ML models that are executed on the platform. The method can also include training the predictor with the instructions or the IRs as learning features and the instruction performance reports as learning labels, compiling the predictor into a library file, and storing the library file in a storage device.