Linear Attention Sequence Parallelism for Long-Sequence GPU Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing sequence parallelism (SP) methods for processing long sequences on a single graphics processing unit (GPU) do not fully utilize linear attention properties, leading to poor parallel efficiency and usability of linear attention-based language models.
Innovation Solution
A linear attention sequence parallelism (LASP) method that distributes subsequences of an original sequence across multiple processing devices, utilizing a preconfigured data distribution strategy, and employs a forward propagation and backpropagation mechanism to determine and update parameters, optimizing communication and processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Length of stationary object
If existing sequence parallelism method is used to process long sequences, then memory limit of single GPU is extended, but parallel efficiency is poor
Solution Approach 1:
The patent divides the original sequence into multiple subsequences and distributes them across different processing devices. Each device processes a specific subsequence independently, enabling parallel computation while maintaining the ability to handle long sequences that exceed single-GPU memory capacity.
Solution Approach 2:
The patent transitions from single-device sequential processing to multi-device parallel processing by introducing a distributed dimension. The sequence parallelism is achieved by distributing subsequences across multiple devices along the sequence dimension, thereby extending the effective sequence length beyond single-device limits while maintaining parallel efficiency.
2Length of stationary object
If existing sequence parallelism method is used, then processing capability for long sequences is improved, but communication overhead is high
Solution Approach 1:
The patent extracts and utilizes the linear attention properties from the transformer architecture to optimize the sequence parallelism implementation. By identifying and leveraging the specific computational patterns in linear attention, the method reduces unnecessary communication operations between devices while maintaining the ability to process long sequences.
Solution Approach 2:
The patent changes the computational parameters and communication patterns to exploit linear attention properties. By modifying how attention computations are performed and communicated across devices, the method reduces communication overhead compared to standard sequence parallelism approaches.
3Device complexity
If linear attention properties are not fully utilized, then implementation is simpler, but parallel efficiency is poor
Solution Approach 1:
The patent performs preliminary analysis and configuration to identify and exploit linear attention properties before executing the sequence parallelism algorithm. By pre-configuring the data distribution strategy and forward propagation method to leverage linear attention characteristics, the system achieves high parallel efficiency without requiring complex runtime adjustments.
Data Source
AI summary
A linear attention sequence parallelism method includes distributing, by a first processing device, each of multiple subsequences corresponding to an original sequence in a distributed environment to a respective one of multiple second processing devices according to a data distribution strategy; determining, by the respective second processing device, a respective one of multiple forward total output matrices corresponding to each subsequence according to a preconfigured forward propagation manner; and determining, by the respective second processing device, a respective one of multiple parameter gradients corresponding to each subsequence according to a preconfigured backpropagation manner and the forward total output matrix, and updating a parameter corresponding to each subsequence according to the respective parameter gradient.