Sparse Mask Matrix Segmentation for Large Language Model Attention Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current large language models face inefficiencies due to high memory occupation and computational overhead from full masking methods, which affect processing efficiency and accuracy.
Innovation Solution
A method is introduced to determine and execute a target attention task based on a sparse representation of mask positions in non-intersecting intervals within a mask matrix, reducing unnecessary calculations by skipping fully-masked regions and optimizing memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full masking method is used in large language models, then attention mechanism can process all regions uniformly, but memory occupation and computational overhead increase significantly
Solution Approach 1:
The patent segments the mask matrix into multiple non-intersecting intervals based on sparse representation, dividing the attention computation into distinct regions (fully-masked, partially-masked, and non-masked intervals). This segmentation allows the system to process only necessary regions separately, avoiding uniform full-mask processing while maintaining attention mechanism completeness.
Solution Approach 2:
The patent extracts and identifies fully-masked intervals from the mask matrix using sparse representation, then excludes these regions from attention computation. By taking out the unnecessary fully-masked regions, the system reduces computational overhead while preserving the attention mechanism's ability to process relevant regions.
2Reliability
If full masking method is used in large language models, then all regions are processed uniformly, but processing efficiency decreases due to unnecessary calculations
Solution Approach 1:
The mask matrix is segmented into multiple non-intersecting intervals identified by sparse representation, allowing the system to distinguish between fully-masked, partially-masked, and non-masked regions. This segmentation enables selective processing of only necessary intervals, improving processing efficiency while maintaining completeness for non-masked regions.
Solution Approach 2:
The patent implements skipping of fully-masked intervals by identifying them through sparse representation and excluding them from attention computation. This allows the system to rush through or skip unnecessary calculations in fully-masked regions while still processing relevant regions, thereby improving overall processing efficiency without sacrificing completeness.
3Loss of energy
If sparse representation with non-intersecting intervals is used, then computational overhead is reduced, but implementation complexity increases
Solution Approach 1:
While segmentation into non-intersecting intervals does increase implementation complexity, the patent manages this by establishing clear rules for identifying and processing different interval types. The segmentation structure itself simplifies the overall logic by providing distinct handling paths for fully-masked, partially-masked, and non-masked regions, making the complexity manageable and structured.
Solution Approach 2:
The patent applies local quality by treating different intervals with different processing qualities - fully-masked intervals are excluded entirely, partially-masked intervals receive selective processing, and non-masked intervals receive full attention computation. This localized processing strategy reduces overall computational overhead while maintaining appropriate processing quality for each region type.
4Device complexity
If traditional masking is used, then simplicity is maintained, but memory occupation increases
Solution Approach 1:
The patent introduces segmentation of the mask matrix into non-intersecting intervals identified by sparse representation. This segmentation reduces memory occupation by avoiding the need to store and process mask information for all regions uniformly, while the segmentation logic itself remains relatively simple to implement through systematic interval identification and classification.
Data Source
AI summary
A method of executing a task for a large language model, a device, and a storage medium are provided, which relate to a field of artificial intelligence technology, and in particular to fields of deep learning, large language model, natural language processing and computer vision technologies. The method includes: determining, by using a determination unit, a target attention task from a plurality of attention tasks to be processed, based on a sparse representation corresponding to a feature to be processed, where the target attention task is a task corresponding to a non-fully masked region of the feature, the sparse representation represents a mask position of the feature, and the mask position represents mask endpoint positions in at least two non-intersecting intervals in a mask matrix corresponding to the feature; and executing the target attention task by using a computing unit, so as to obtain an attention feature.