Tensor Processing for Variable Sequence Lengths
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
BERT-like models consume increasing parameter space and computational resources, leading to high training and inference costs, and existing solutions fail to efficiently handle variable-length inputs in Transformer models, resulting in redundant computations and memory overhead.
Innovation Solution
The method involves dividing input tensors into matrices based on the number of inputs and attention heads, applying grouped matrix multiplication with shared memory, and using a padding-free algorithm to avoid computations on useless tokens, optimizing multi-head attention through kernel fusion and grouped GEMM to reduce redundant calculations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If BERT-like models use larger parameter space to improve performance, then model accuracy is improved, but computational resources and training costs increase
Solution Approach 1:
The patent segments the batch of inputs with variable sequence lengths into multiple sub-batches, each with uniform sequence lengths. This segmentation allows independent processing of each sub-batch, eliminating the need to pad shorter sequences to the maximum length, thereby reducing redundant computations while maintaining model accuracy.
Solution Approach 2:
The patent dynamically adjusts the processing strategy based on the actual sequence lengths in the batch. By identifying and processing sequences of different lengths separately rather than using a fixed maximum length for all, the system adapts computational resources to actual needs, reducing waste without compromising performance.
2Ease of operation
If padding is used to handle variable-length inputs, then processing simplicity is improved, but redundant computations on useless tokens increase
Solution Approach 1:
Instead of padding all sequences to a fixed length, the patent segments the batch into sub-batches where each sub-batch contains sequences of the same length. This eliminates the need for padding within each sub-batch while maintaining the simplicity of batch processing, as each segment can be processed independently without redundant operations.
Solution Approach 2:
The patent changes the parameter of sequence length from a fixed maximum value to variable values that match the actual input lengths. By processing sequences with their true lengths rather than padded lengths, the system eliminates computations on nonexistent tokens while preserving processing efficiency.
3Productivity
If fixed sequence length is used for batch processing, then processing efficiency is improved, but adaptability to variable-length inputs deteriorates
Solution Approach 1:
The patent combines fixed-length processing efficiency with variable-length adaptability by segmenting the variable-length batch into multiple fixed-length sub-batches. Each sub-batch maintains the processing efficiency of fixed-length operations while the overall system adapts to variable-length inputs through the segmentation strategy.
Solution Approach 2:
The system dynamically determines the segmentation strategy based on the actual sequence lengths in the input batch. By adjusting how the batch is divided into sub-batches according to the varying lengths, the system maintains high processing efficiency for each sub-batch while adapting to the overall variable-length nature of the input data.
Data Source
AI summary
A method is proposed for tensor processing. A first tensor is obtained to be applied attention, the first tensor representing a batch of inputs with variable sequence lengths. The first tensor is divided into a plurality of matrices based on the number of inputs of the batch and the number of heads of the attention, wherein a matrix of the plurality of matrices has a dimension corresponding to the sequence length of an input of the batch and a dimension corresponding to a head size of the attention. A second tensor is generated by applying the attention to the plurality of matrices respectively based on a grouped matrix multiplication, wherein a resulting matrix of the grouped matrix multiplication comprises a plurality of tiles each of which is computed by a set of threads with a shared memory. Therefore, the computations on useless tokens are avoided.