Lower Triangular Matrix Allocation for Mask Attention Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for calculating mask attention score matrices in transformer-based artificial neural networks are inefficient due to unnecessary calculations in matrix multiplication, particularly for elements outside the lower triangular matrix.
Innovation Solution
The proposed solution involves optimizing the matrix multiplication calculation by allocating data such that calculations corresponding to masked locations are avoided, effectively replacing zero calculations with non-zero valued elements and focusing on the lower triangular matrix for the result.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional matrix multiplication is used for mask attention score calculation, then complete matrix computation is performed, but computational efficiency deteriorates due to unnecessary zero calculations
Solution Approach 1:
The patent extracts and processes only the lower triangular matrix elements from the complete matrix multiplication result. By separating the useful computation (lower triangular elements needed for mask attention) from the useless computation (upper triangular elements that will be masked), the system eliminates unnecessary calculations and improves computational efficiency.
Solution Approach 2:
The patent applies masking operations in advance during the matrix multiplication process itself, rather than performing complete multiplication and then masking afterward. By preliminarily identifying which elements will be masked and excluding them from computation, the system avoids wasting computational resources on calculations that will ultimately be discarded.
2Measurement precision
If complete matrix multiplication is performed, then all elements are calculated, but calculation time increases due to processing elements outside the lower triangular matrix
Solution Approach 1:
The patent extracts only the lower triangular elements from the matrix multiplication result for further processing. By separating these essential elements from the rest of the matrix, the system performs calculations only where needed, maintaining accuracy for relevant computations while eliminating time-wasting operations on irrelevant upper triangular elements.
Solution Approach 2:
The patent performs partial matrix multiplication by focusing exclusively on the lower triangular portion of the result matrix. This partial action approach calculates only the necessary elements (lower triangular) without performing excessive calculations on the upper triangular elements that would not contribute to the final mask attention score.
3Productivity
If data is allocated for complete matrix computation, then all matrix elements are processed, but resource utilization decreases due to handling masked locations
Solution Approach 1:
The patent extracts and processes only the lower triangular matrix elements from the complete matrix computation result. By separating the essential computation from the redundant computation, the system optimizes data allocation to handle only necessary elements, reducing the complexity of data management while improving processing throughput.
Data Source
AI summary
A computer implementation method for obtaining a lower triangular matrix for a matrix multiplication result value, the method comprising: a process of allocating elements of each of a first operand matrix pair and a second operand matrix pair input into a memory to calculation units of an accelerator; and a process of the calculation units of the accelerator performing a matrix multiplication calculation on the elements of the first operand matrix pair and the second operand matrix pair, wherein the process of allocating the elements thereof to the calculation units of the accelerator comprises: allocating elements of the first operand matrix pair to first calculation units corresponding to elements of the lower triangular matrix for the matrix multiplication result value of the first operand matrix pair; and allocating elements of the second operand matrix pair to second calculation units corresponding to elements of the lower triangular matrix for the matrix multiplication result value of the second operand matrix pair.