Hybrid Self-Attention for Low-Latency Decoder Token Generation

The patent segments the self-attention computation into two distinct stages: context stage and token generation stage. In the context stage, sparse attention is applied where each token only attends to a limited number of previously identified tokens (e.g., L=8), reducing computational complexity from O(N^2) to O(NL). In the token generation stage, full self-attention is restored to ensure high-quality output generation. This segmentation allows the system to maintain output quality while achieving real-time processing efficiency.

The patent applies local quality by using different attention mechanisms in different stages of the decoding process. The context stage uses sparse attention with local quality optimization for efficiency, while the token generation stage uses full self-attention to ensure high precision attention accuracy when generating tokens. This staged approach allows each stage to have the appropriate level of attention accuracy needed for its specific function.

The patent applies preliminary action by performing sparse attention computation in the context stage before token generation. This preliminary sparse attention establishes efficient context representations with reduced computational overhead, preparing the model for subsequent accurate token generation. By doing the efficiency-optimized work first, the system reduces overall latency while maintaining the capability for accurate attention when needed in the generation stage.