Transformer Model Inference Compression via Activation Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformer-based natural language processing models are computationally expensive due to self-attention mechanisms, with redundant processing of words across transformer blocks, especially for tasks where not all words are equally important, such as stop words and verbs/identifiers in entity recognition.
Innovation Solution
The technique involves inserting soft and hard extraction/introduction layers into transformer blocks to determine the significance of word vectors, selectively processing only the most important words, and removing redundant output activations, thereby compressing the model while maintaining accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If self-attention mechanisms are applied to process all words through transformer blocks, then model accuracy is maintained, but computational cost and processing time increase significantly
Solution Approach 1:
The patent extracts and removes redundant output activations from transformer blocks during inference. By identifying and eliminating unnecessary computational operations (particularly for stop words and less important tokens), the system reduces processing time while maintaining model accuracy for the remaining significant words.
Solution Approach 2:
The patent applies partial action by selectively processing only the most important words through all transformer blocks, while applying reduced or skipped processing to less important words. This partial processing strategy maintains accuracy for critical tokens while improving overall inference speed.
2Reliability
If all words are processed through each transformer block, then complete language understanding is achieved, but runtime memory usage increases
Solution Approach 1:
The patent extracts redundant intermediate activations from the transformer block processing pipeline. By removing these unnecessary memory-stored activations (particularly for less important tokens), the system reduces runtime memory consumption while preserving the computational path for significant words that require complete processing.
Solution Approach 2:
The patent applies different processing qualities to different parts of the input sequence. Important words receive full transformer block processing with all activations retained, while less important words receive reduced processing with activations discarded, optimizing the balance between understanding completeness and memory usage.
3Productivity
If redundant output activations are removed from transformer blocks, then inference speed increases, but model complexity increases due to compression
Solution Approach 1:
The patent performs preliminary analysis during the training phase to identify which words and activations are redundant for specific tasks and datasets. This pre-computed knowledge is stored and applied during inference to guide the removal of redundant activations, avoiding the need for complex real-time analysis and simplifying the inference process.
Solution Approach 2:
The system uses its own training data and task requirements to automatically identify and remove redundant activations. The model self-adapts to the specific NLP task by analyzing which computational paths are unnecessary, eliminating the need for external manual optimization and reducing overall system complexity.
Data Source
AI summary
Methods, systems, and computer program products for accelerating inference of transformer-based models are provided herein. A computer-implemented method includes obtaining a machine learning model comprising a plurality of transformer blocks, a task, and a natural language dataset; generating a compressed version of the machine learning model based on the task and the natural language dataset, wherein the generating comprises: obtaining at least one set of tokens, wherein each token in the set corresponds to one of the items in the natural language dataset, identifying and removing one or more redundant output activations of different ones of the plurality of transformer blocks for the at least one set of tokens, and adding one or more input activations corresponding to the one or more removed output activations into the machine learning model at subsequent ones of the plurality of the transformer blocks; and outputting the compressed version of the machine learning model to at least one user.