Transformer Attention Row Clustering for Lower Memory Complexity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformers require exponentially increasing memory resources due to the square of the number of input data for attention operations, leading to high computational complexity.
Innovation Solution
Apply low rank approximation to query and key vectors, cluster rows based on similarity, determine representative rows, and create a sub-query vector to calculate sub-attention values, reducing data dimension and memory capacity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If attention operation is performed using all input data, then processing accuracy is improved, but memory capacity and computational complexity increase exponentially
Solution Approach 1:
The patent segments the input data by applying low rank approximation to decompose the query or key matrix into smaller factor matrices. This segmentation reduces the dimensionality of the attention computation from O(N) to O(k) where k << N, thereby decreasing computational complexity while preserving the essential patterns needed for accurate processing
Solution Approach 2:
The patent creates approximate representations (copies) of the original data through low rank approximation. Instead of processing all original input data, it processes compressed approximate versions that capture the dominant patterns, reducing memory requirements and computational load while maintaining sufficient accuracy for the attention operation
2Measurement precision
If attention operation is performed using all input data, then processing accuracy is improved, but memory capacity increases exponentially
Solution Approach 1:
The patent segments the large input data into compressed representations through low rank approximation. By decomposing matrices into smaller factor matrices with rank k, it reduces the memory capacity from O(N^2) to O(Nk) where k is much smaller than N, enabling processing of large datasets with limited memory resources
Solution Approach 2:
The patent creates compact approximate copies of the original data using low rank decomposition. These compressed copies retain the essential information needed for attention computation while occupying significantly less memory space, allowing accurate processing without exponential memory requirements
3Device complexity
If row clustering is applied to reduce data dimension, then computational complexity is decreased, but processing accuracy may be reduced
Solution Approach 1:
The patent changes the parameter of data representation from full-rank to low-rank approximation. By adjusting the rank parameter k to be much smaller than the original dimension N, it achieves a balance between computational complexity (reduced by factor of N/k) and processing accuracy (maintained through preservation of dominant singular vectors that capture essential data patterns)
Data Source
AI summary
The present disclosure is characterized by reducing the data dimension of an attention operation through row clustering on the basis of the fact that most rows constituting an attention score of a transformer have similar patterns.