Transformer Attention Blocks With Skipped Key Embedding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformer models require significant computational resources for training and inference, which can be inefficient and hinder their widespread application, especially in resource-constrained devices.
Innovation Solution
The method involves a transformer model architecture that includes patch embedding and attention blocks, where key and value embedding computations are skipped, and spatial reduction is applied, reducing computational complexity while maintaining performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If transformer models use standard attention mechanisms with full embedding computations, then model performance is maintained, but computational resources and training time increase significantly
Solution Approach 1:
The patent extracts and removes the key embedding computation step from the standard attention mechanism. By skipping the key embedding computation and only performing query and value embeddings, the model reduces computational complexity while maintaining performance through alternative attention weight calculation methods
Solution Approach 2:
The patent applies partial action by performing only essential embedding computations (query and value) while omitting less critical ones (key embedding). This selective computation approach reduces overall computational burden while preserving the necessary information for accurate attention mechanisms
2Measurement precision
If transformer models perform complete embedding computations for all attention components, then inference accuracy is maintained, but device complexity and computation time increase
Solution Approach 1:
The patent extracts the key embedding computation from the inference process. By removing this step and using alternative methods to calculate attention weights, the system achieves faster inference while maintaining accuracy through optimized attention mechanisms that rely on query and value embeddings alone
3Reliability
If transformer models use standard attention block computations, then model performance is preserved, but the model size and computation quantities increase
Solution Approach 1:
The patent removes the key embedding computation layer from the attention block structure. This extraction simplifies the model architecture by eliminating redundant computational steps while maintaining performance through reformed attention mechanisms that compute attention weights directly from query and value embeddings
Solution Approach 2:
The patent implements partial action by performing only the essential embedding computations (query and value) rather than all three standard embeddings (query, key, and value). This reduces model complexity and computation quantities while preserving necessary functional capabilities
Data Source
AI summary
Provided is a method of performing inference by using a transformer model including a plurality of encoders and a plurality of decoders, wherein each of the plurality of encoders and the plurality of decoders includes a transformer block including an attention block, and the method is performed by an electronic device and includes receiving input data and using the transformer model to perform inference on the input data, thereby generating output data, wherein the generating of the output data includes skipping performing a key embedding computation, a query embedding computation, and a value embedding computation on input data of the attention block.