Adaptive Off-Ramp Training for Early Exit in Deep Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks with increasing number of layers face higher latency in generating inferences, which is a challenge in real-time systems, and existing attempts to improve performance result in tradeoffs between accuracy, complexity, and training time.
Innovation Solution
Training 'off-ramps' on respective prediction layers of a deep neural network for adaptive exits, allowing early exit when a label predicted by an intermediate representation layer has a sufficient confidence value, thereby reducing the need for subsequent layers to process the input.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the number of layers in a deep neural network is increased to improve accuracy, then prediction accuracy is improved, but latency increases
Solution Approach 1:
The neural network is segmented into multiple prediction layers, each capable of independently making predictions. The system divides the deep network into stages where early exiting is possible, allowing the input to be processed only as deep as necessary based on confidence thresholds, thus reducing latency while maintaining accuracy
Solution Approach 2:
The system dynamically adjusts the number of layers processed based on confidence values. When a prediction layer achieves sufficient confidence, the system exits early rather than processing all layers. This dynamic adaptation allows the network to optimize between accuracy and latency for each specific input case
2Loss of time
If early exiting is implemented to reduce latency, then processing time is reduced, but prediction accuracy may deteriorate
Solution Approach 1:
The system implements feedback through confidence value evaluation at each prediction layer. The confidence values are fed back to determine whether to exit early or continue processing. This feedback mechanism ensures that early exiting occurs only when sufficient accuracy is achieved, balancing processing time and prediction accuracy
Solution Approach 2:
The system changes the parameter of processing depth dynamically based on confidence thresholds. By adjusting which layers are processed and under what conditions, the system optimizes the trade-off between processing time and accuracy for different input scenarios
3Speed
If quantization is used to compress the neural network for faster processing, then processing speed is improved, but model robustness deteriorates
Solution Approach 1:
Instead of compressing the entire network through quantization, the system segments the network into prediction layers with off-ramps. This allows the full precision network to be used selectively only where necessary, maintaining robustness while improving speed through early exiting for simple inputs
4Loss of time
If distillation training is used to create a smaller network, then processing time is reduced, but the complexity of training procedures increases
Solution Approach 1:
The system performs preliminary training of off-ramps at each prediction layer using a two-stage process. During training, data is routed to appropriate layers based on confidence thresholds, and off-ramps are pre-trained to handle specific prediction tasks. This preliminary preparation enables fast inference without complex training procedures during deployment
Data Source
AI summary
Systems and methods are provided for training and using a deep neural network with adaptively trained off-ramps for an early exit at an intermediate representation layer. The training includes, for respective intermediate representation layers of a sequence of intermediate representation layers, predicting a label based on the training data and comparing against a correct label. The training further includes generating a confidence value associated with the predicted label. The confidence value is based on optimizing an objective function that includes a weighted entropy of a probability distribution of the likelihood, weighted based on whether previous intermediate representation layer has accurately predicted the label. Use of the weighted entropy provides the training with a focus on predicting labels that the previous intermediate representation layers has performed poorly and not labels that have existed before the intermediate representation layer being trained. Among alternative methods include a distilled twin, parallel neural network for predicting labels using adaptively trained off-ramps.