Early Exiting Transformer Layers Using Entropy Confidence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current pre-trained language models face issues with overthinking and high latency, leading to user inconvenience, particularly in scenarios like online medical consultations, due to inflexible speed improvement ratios in existing early exiting methods.
Innovation Solution
The proposed solution involves an electronic device that calculates a probability distribution for prediction results across multiple transformer layers, measures confidence levels using entropy values, and terminates inference early when a predefined confidence level is met in consecutive layers, allowing for flexible adjustment of the speed improvement ratio.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If early exiting is implemented with fixed parameters, then inference speed is improved, but the speed improvement ratio becomes inflexible and cannot be adjusted
Solution Approach 1:
The patent applies dynamics by making the early exiting parameters adjustable rather than fixed. The system allows dynamic configuration of the number of middle layers and confidence level thresholds, enabling the speed improvement ratio to be flexibly adjusted according to different application scenarios while maintaining inference speed improvements.
Solution Approach 2:
The patent changes parameters by introducing configurable parameters (number of middle layers, confidence level threshold) that can be modified to adjust the early exiting behavior. This allows the system to adapt the inference speed and accuracy trade-off by changing these parameters without requiring model retraining or structural modifications.
2Reliability
If deep transformer layers are used for accurate prediction, then prediction accuracy is improved, but latency increases causing user inconvenience
Solution Approach 1:
The patent applies partial action by allowing the inference process to terminate early at middle layers when confidence thresholds are met, rather than always processing through all deep transformer layers. This partial processing maintains accuracy for confident predictions while reducing latency for cases where full processing is unnecessary.
Solution Approach 2:
The patent segments the transformer model into multiple middle layers with early exiting points, allowing the inference process to be divided into multiple potential termination points. This segmentation enables the system to achieve accurate predictions at shallower depths when possible, reducing latency while maintaining the option to go deeper if needed.
3Loss of time
If early exiting is implemented, then latency is reduced, but the ability to handle complex samples that require deeper processing is compromised
Solution Approach 1:
The patent applies dynamics by making the early exiting decision adaptive rather than static. The system dynamically determines whether to exit early based on confidence level measurements at each middle layer, allowing complex samples that require deeper processing to automatically continue through more layers while simple samples terminate early, thus maintaining accuracy for complex cases while reducing latency overall.
Solution Approach 2:
The patent uses feedback by measuring confidence levels at each middle layer and using this information to decide whether to continue processing or exit early. This feedback mechanism ensures that samples with low confidence (likely complex samples) receive additional processing layers to improve accuracy, while high-confidence samples can terminate early to reduce latency.
Data Source
AI summary
The electronic device for improving the inference performance of a pre-trained language model according to an exemplary embodiment of the present invention includes a processor for sequentially passing input data through a plurality of transformer layers of the pre-trained language model and obtaining output data, wherein the processor calculates a probability distribution for prediction results received from each transformer layer in each of a plurality of middle layers connected to each rear end of the plurality of transformer layers, and measures a confidence level of the prediction results based on an entropy value calculated by using the probability distribution, and wherein when a confidence level less than a predefined value is measured in a predefined number of consecutive middle layers among the plurality of middle layers, the processor outputs a prediction result of the first middle layer as the output data by taking the last first middle layer among the consecutive middle layers as an exit.