Adaptive Token Depth Adjustment in Transformer Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformer neural networks incur high latency and resource overhead due to computationally intensive matrix multiplication operations, limiting their use in devices with limited computational capabilities, memory, power, and network bandwidth.
Innovation Solution
A technique is introduced where halting scores are computed for each token at each layer of the transformer neural network, and if the score exceeds a threshold, processing of that token is halted, reducing the number of tokens processed and enabling more efficient execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If transformer neural networks process all tokens through all layers, then accuracy is improved, but computational cost and latency increase quadratically
Solution Approach 1:
The patent applies dynamic adaptive token depth adjustment where the processing depth for each token is dynamically determined based on its importance score. Tokens with lower importance scores are processed by fewer layers, while tokens with higher importance scores undergo complete processing through all layers. This dynamic approach allows the system to adaptively allocate computational resources, reducing overall computational cost while maintaining accuracy for critical tokens.
Solution Approach 2:
The patent implements local quality by applying different processing depths to different tokens within the same input sequence. Instead of uniformly processing all tokens through all layers, the system selectively applies varying levels of processing based on local importance assessments. This allows computationally intensive processing only where necessary, improving overall efficiency while preserving accuracy for important tokens.
2Reliability
If transformer neural networks process all tokens through all layers, then task performance is improved, but resource overhead and power consumption increase
Solution Approach 1:
The system dynamically adjusts the processing depth for each token based on its importance score, enabling adaptive resource allocation. Tokens deemed less important are processed by fewer layers, reducing computational operations and corresponding power consumption. This dynamic adaptation allows the system to maintain task performance while significantly reducing energy usage compared to uniform full-depth processing.
Solution Approach 2:
The patent changes the processing parameter (number of layers) based on the importance score parameter of each token. By varying the processing depth parameter dynamically according to token-specific importance metrics, the system optimizes the balance between task performance and energy consumption, avoiding unnecessary computational operations on less critical tokens.
3Reliability
If transformer neural networks process all tokens through all layers, then accuracy is improved, but memory and network bandwidth requirements increase
Solution Approach 1:
The system dynamically determines the processing depth for each token based on importance scores, enabling adaptive memory and bandwidth allocation. Tokens with lower importance require less memory access and network bandwidth, while critical tokens receive full processing resources. This dynamic resource allocation reduces overall memory and bandwidth requirements while preserving accuracy for important tokens.
Solution Approach 2:
The patent applies local quality by providing different levels of processing depth to different tokens based on their importance. This selective approach reduces the aggregate memory access and network bandwidth requirements by limiting full processing only to tokens that truly require it, thereby reducing resource consumption while maintaining overall accuracy.
4Productivity
If transformer neural networks use adaptive token depth adjustment, then computational efficiency is improved, but system complexity increases
Solution Approach 1:
The system performs preliminary computation of importance scores for tokens before determining processing depth. By pre-assessing token importance and using these scores to guide subsequent processing decisions, the system avoids the need for complex real-time adjustments during the main processing phase. This preliminary action simplifies the overall system architecture while maintaining computational efficiency benefits.
Data Source
AI summary
One embodiment of the present invention sets forth a technique for executing a transformer neural network. The technique includes computing a first set of halting scores for a first set of tokens that has been input into a first layer of the transformer neural network. The technique also includes determining that a first halting score included in the first set of halting scores exceeds a threshold value. The technique further includes in response to the first halting score exceeding the threshold value, causing a first token that is included in the first set of tokens and is associated with the first halting score not to be processed by one or more layers within the transformer neural network that are subsequent to the first layer.