Transformer Model Softmax Tile-Wise Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformer models face performance bottlenecks due to memory-intensive softmax operations, which occupy significant processing time and increase memory access, limiting their efficiency in tasks like natural language processing.
Innovation Solution
The implementation of a transformer model with a processor that decomposes the softmax operation into a first sub-softmax operation, a reduction operation, and a second sub-softmax operation, allowing these operations to be fused with matrix multiplication operations performed in a tile-wise manner, reducing memory access and resource consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If softmax operations are performed separately from matrix multiplication, then calculation accuracy is maintained, but memory access increases and processing time increases
Solution Approach 1:
The patent merges the softmax operation with matrix multiplication operations by performing tile-wise softmax calculations during the matrix multiplication process. The attention block computes softmax values for tiles while simultaneously performing matrix multiplication, eliminating separate memory access operations and reducing processing time while maintaining calculation accuracy through precise tile-wise computation.
2Device complexity
If softmax operations are performed separately from matrix multiplication, then computational steps are clear and simple, but memory access increases and resource consumption increases
Solution Approach 1:
The patent combines multiple operations (matrix multiplication and softmax) into a single fused operation that processes tiles together. This merging reduces the number of separate computational passes required, decreasing memory access frequency and energy consumption while maintaining the logical clarity of the computational steps through structured tile-wise processing.
3Use of energy by moving object
If tile-wise operations are used for softmax, then memory access is reduced, but operational complexity increases
Solution Approach 1:
The patent segments the attention block computation into tile-wise operations, dividing the matrix into smaller tiles that can be processed independently. This segmentation reduces memory access by keeping computations localized to tile boundaries while managing operational complexity through systematic tile processing routines that handle softmax and matrix multiplication together for each tile.
4Ease of operation
If separate softmax and matrix multiplication operations are used, then operational clarity is maintained, but processing efficiency decreases
Solution Approach 1:
The patent merges softmax and matrix multiplication into a single fused operation that processes tiles together in the attention block. This merging improves processing efficiency by eliminating redundant memory access and computational overhead while maintaining operational clarity through structured tile-wise processing that keeps the computational logic organized and manageable.
Data Source
AI summary
A device and method with transformer model implementation are provided. The electronic device includes a processor configured to perform an inference by implementing a transformer model including a plurality of encoders and a plurality of decoders, and a memory configured to store instructions to be executed by the processor. Each of the encoders and the decoders includes an attention block that determines an attention value. The processor is configured to perform a first sub-softmax tile-wise operation in the attention block, perform a reduction operation to determine an adjustment factor based on a resulting value of the first sub-softmax operation, and perform a second sub-softmax tile-wise operation based on a resulting value of the reduction operation.