Dynamic Neural Network Compression via Early Exit Segments
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning model compression techniques fail to dynamically adapt to real-time demands, leading to inefficient resource usage and power consumption, as they do not consider input data difficulty and require multiple fixed model variants for varying performance requirements.
Innovation Solution
A method that divides deep neural networks into segments with early exits, allowing dynamic configuration of compression based on target accuracy and complexity, enabling flexible performance tuning and reducing computational overhead by exiting early when confidence thresholds are met.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple fixed model variants are created to support different compression rates, then performance requirements can be met, but storage space increases and model switching becomes difficult
Solution Approach 1:
The patent implements dynamic model compression by introducing a compression controller that adjusts the compression rate of the deep neural network in real-time based on current performance requirements and resource conditions. Instead of managing multiple fixed model variants, the system dynamically modifies the compression rate of a single base model, allowing flexible adaptation to different performance targets without the overhead of storing and switching between multiple model versions.
2Ease of manufacture
If fixed data path models are used, then implementation is simple, but computation is wasted on easy inputs and entire models must be replaced when performance targets change
Solution Approach 1:
The patent implements dynamic model compression by introducing a compression controller that adjusts the compression rate of the deep neural network in real-time based on current performance requirements and resource conditions. Instead of managing multiple fixed model variants, the system dynamically modifies the compression rate of a single base model, allowing flexible adaptation to different performance targets without the overhead of storing and switching between multiple model versions.
3Reliability
If deep models are used to handle difficult input cases, then acceptable results are obtained, but computation time and power consumption increase
Solution Approach 1:
The patent applies partial action by dynamically adjusting the compression rate based on input difficulty. For easy input cases, a higher compression rate (lower model complexity) is applied since full model capacity is not needed. For difficult cases, the compression rate is reduced to maintain accuracy. This selective application of model complexity avoids wasting computation and energy on easy cases while ensuring sufficient capacity for difficult cases.
Solution Approach 2:
The compression controller dynamically adjusts the compression rate in real-time based on performance requirements and resource conditions, allowing the model to adapt its complexity level to match the actual demand of each inference task, thereby optimizing the trade-off between accuracy and energy consumption.
Data Source
Figure 1A
Figure 1B
Figure 2
AI summary
Disclosed is a method of dynamically compressing and scaling a deep neural network, that includes dividing the deep neural network into a plurality of segments, and a final early exit connected to a last segment of the plurality of segments, connecting at least one early exit to an output of each segment during run-time, based on an early exit configuration that is determined dynamically during the run-time based on a target accuracy and complexity, and ending the run-time and exiting the deep neural network from an early exit, and setting an output of said early exit as a final output of the deep neural network, when said early exit reports a confidence value higher than an associated threshold.