Runtime Performance Estimator for DNN Model Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing model compression methods for deep neural networks (DNNs) rely on proxy signals that do not accurately reflect runtime performance on different hardware architectures, leading to suboptimal compression strategies.
Innovation Solution
The method involves determining an initial DNN and its associated compression algorithm, emulating different compressed models on target hardware to obtain runtime performance data, and using regression to create a runtime performance estimator, which is then applied to the compression algorithm to generate a compressed DNN that meets specific performance requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If proxy signals (FLOPs, model size) are used to guide compression, then compression speed is improved, but compression accuracy does not reflect actual runtime performance
Solution Approach 1:
The patent introduces a runtime performance estimator as an intermediary component that bridges the gap between proxy signals and actual runtime performance. This estimator uses regression models trained on hardware-specific data to translate compression configurations into accurate runtime performance predictions, resolving the contradiction between fast compression and accurate performance assessment.
Solution Approach 2:
The system performs preliminary hardware-specific regression model training and emulation before actual compression. By pre-establishing the relationship between compression parameters and runtime performance through emulation on target hardware, the system prepares accurate prediction models in advance, enabling both fast and accurate compression without sacrificing prediction precision.
2Measurement precision
If hardware-specific emulation is performed to obtain accurate runtime performance data, then runtime performance prediction accuracy is improved, but computation time increases
Solution Approach 1:
The patent performs hardware-specific emulation and regression model training as a preliminary one-time setup process before actual compression operations. This upfront investment in accurate hardware characterization enables subsequent compression operations to use fast regression-based predictions without repeated expensive emulation, resolving the time-accuracy tradeoff.
Solution Approach 2:
The system creates a simplified regression-based copy of the actual hardware runtime behavior that can be evaluated quickly. Instead of repeatedly running expensive hardware emulation, the patent uses a learned regression model that replicates hardware performance characteristics, enabling fast and accurate compression decisions without repeated time-consuming emulation.
3Reliability
If compression is optimized for one hardware architecture, then performance on that hardware is improved, but adaptability to different hardware platforms deteriorates
Solution Approach 1:
The patent applies local quality by creating hardware-specific regression estimators tailored to each target platform's characteristics. Instead of using a generic compression approach, the system adapts the performance model to local hardware properties (CPU, GPU, FPGA, mobile processors), achieving optimal performance on each platform while maintaining the ability to adapt to new hardware through the same framework.
Solution Approach 2:
The system achieves universality through a unified regression-based framework that can be adapted to multiple hardware platforms. The same compression engine and methodology work across different hardware types (CPU, GPU, FPGA, mobile), with platform-specific regression models providing the necessary adaptation. This allows the system to serve multiple hardware targets while maintaining consistent compression quality.
Data Source
AI summary
A method, computer system and computer program product for model compression service. The method comprises determining an initial deep neural network (DNN) and an associated compression algorithm available in a compression engine, a type of target hardware and a performance requirement of target hardware. The method also comprises emulating a plurality of different compressed models of the initial DNN on target hardware of the type to obtain corresponding runtime performance data, wherein the different compressed models are defined with different configuration data. The method further comprises obtaining a runtime performance estimator of the target hardware by regression with the different configuration data and the corresponding runtime performance data. Lastly, the method comprises applying the runtime performance estimator to the compression algorithm by the compression engine to generate a compressed DNN of the initial DNN complying with the performance requirement of the type of target hardware.