Attention Propagation in Transformer Blocks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional vision transformers are computationally expensive due to self-attention computation at every block, limiting their applicability to low-power devices despite their accuracy and effectiveness.
Innovation Solution
Propagating self-attention output from one transformer block to subsequent blocks, reducing redundant computation by reusing attention information, which is dynamically modified using operations like convolution, thereby reducing computational expense while maintaining accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If self-attention is computed at every transformer block, then model accuracy is improved, but computational expense and memory usage increase quadratically
Solution Approach 1:
The patent applies preliminary action by computing self-attention only at selected transformer blocks (e.g., every Nth block or at specific layers) rather than at every block. The attention outputs from these selected blocks are then propagated forward through subsequent blocks, eliminating the need to recompute attention at intermediate blocks while maintaining model accuracy.
Solution Approach 2:
The patent merges the self-attention computation function across multiple transformer blocks by sharing attention outputs. Instead of each block independently computing self-attention, the system combines the attention computation at selected blocks and propagates these combined attention outputs to multiple downstream blocks, reducing redundant computation.
2Measurement precision
If self-attention is computed at every transformer block, then model accuracy is improved, but memory usage increases quadratically with input size
Solution Approach 1:
The patent computes attention outputs in advance at selected transformer blocks and stores these precomputed attention maps. Subsequent blocks reuse these precomputed attention outputs through propagation, avoiding the need to allocate quadratic memory for attention computations at every block while preserving the attention information needed for accurate predictions.
3Productivity
If attention propagation is implemented, then computational burden is reduced, but model complexity increases
Solution Approach 1:
The patent introduces attention propagation as an intermediary mechanism between selected transformer blocks. This propagation layer acts as a mediator that transfers attention outputs from computation blocks to subsequent blocks, simplifying the overall architecture by eliminating redundant self-attention sub-blocks while maintaining the necessary information flow for accurate predictions.
Data Source
AI summary
Certain aspects of the present disclosure provide techniques and apparatus for improved attention-based machine learning. A first attention propagation output is generated using a first transformer block of a plurality of transformer blocks, this generation including processing input data for the first transformer block using a first self-attention sub-block of the first transformer block. The first attention propagation output is propagated to a second transformer block of the plurality of transformer blocks. An output for the second transformer block is generated, this generation including generating output features for the second transformer block based on the first attention propagation output.