Attention Head Sparsity Patterns for Faster Long-Prompt Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The computational challenges of Large Language Model (LLM) inference, particularly with long prompts, are hindered by the quadratic complexity of attention computation, leading to significant latency in the pre-filling stage, which affects the efficiency and accuracy of model outputs.
Innovation Solution
A technique called MInference is introduced, which identifies dynamic sparse attention patterns (A-shape, vertical-slash, and block-sparse) for transformer models, reducing the number of floating-point operations (FLOPs) by computing only the highest-relevance portions of attention weights through a kernel-aware search method.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If dense attention computation is used in transformer models, then attention accuracy is maintained, but pre-filling latency increases significantly with long prompts
Solution Approach 1:
The patent segments the attention computation by identifying and computing only the most relevant attention scores above a threshold, rather than computing all attention scores. This segmentation of computation based on relevance threshold allows maintaining accuracy for important attention relationships while skipping unnecessary computations, thereby reducing pre-filling latency significantly with long prompts
Solution Approach 2:
The patent applies partial action by computing only a subset of attention scores that exceed a relevance threshold, rather than computing all attention scores. This partial computation approach maintains sufficient accuracy for the most important attention relationships while dramatically reducing the computational burden and pre-filling latency, especially for long context sequences
2Adaptability or versatility
If the number of attention heads is increased, then model capacity improves, but computational complexity increases quadratically
Solution Approach 1:
The patent extracts and computes only the most relevant attention scores above a threshold, removing unnecessary computations from the attention mechanism. This extraction approach allows the model to maintain capacity with multiple attention heads while reducing the quadratic computational complexity by focusing only on significant attention relationships
Solution Approach 2:
The patent applies local quality by using different computation strategies for different parts of the attention matrix - computing scores above a threshold with high precision while using approximations or skipping computations for scores below the threshold. This allows maintaining model capacity while reducing overall computational complexity
3Length of stationary object
If prompt length is increased, then context coverage improves, but pre-filling time increases due to quadratic complexity
Solution Approach 1:
The patent performs preliminary action by computing attention scores in a staged manner, first identifying scores above a relevance threshold before performing full computation. This preliminary filtering allows the model to handle longer prompts efficiently by pre-identifying which attention computations are necessary, thereby reducing pre-filling time while maintaining context coverage
Solution Approach 2:
The patent applies dynamics by adapting the attention computation strategy based on the input characteristics and relevance threshold. The computation approach dynamically adjusts between full attention and sparse attention based on the specific prompt and context, allowing efficient handling of varying prompt lengths while maintaining accuracy when needed
Data Source
AI summary
A computing system including processing circuitry configured to, during a calibration stage, perform a sparsity pattern search on a plurality of attention heads included in one or more transformer layers to select a respective sparsity pattern associated with each of the attention heads. During an inferencing stage, processing circuitry receives an inferencing input. The processing circuitry pre-fills a context based at least in part on the inferencing input. Pre-filling the context includes computing sparse attention scores at each of the attention heads. Computing the sparse attention scores includes masking each of the attention heads using the respective sparsity pattern selected for that attention head during the calibration stage. The processing circuitry computes an inferencing output by performing inferencing starting from the sparse attention scores. The processing circuitry outputs the inferencing output.