Pipeline-parallel Dataflow System for Self-Attention Acceleration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analog AI accelerators face performance limitations due to the self-attention bottleneck, particularly in natural language processing, where self-attention computations grow quadratically with sequence length, making it challenging to achieve efficient and energy-effective processing.
Innovation Solution
A pipeline-parallel-dataflow system is implemented to accelerate self-attention computations by delaying the division operation in softmax computations, using pipelined hardware with closely connected SRAM for efficient access to Key, Query, and Value matrices, and performing matrix multiplications in stages to optimize throughput and energy efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If self-attention computations are performed using traditional analog AI accelerators, then matrix operations can be executed, but the system suffers from quadratic complexity growth with sequence length and poor performance on activation matrix multiplications
Solution Approach 1:
The self-attention computation is divided into distinct pipeline stages: QK stage for attention score computation, softmax stage for normalization, and PV stage for value transformation. Each stage processes data independently and passes results to the next stage, enabling parallel execution and reducing overall computational complexity from quadratic to linear scaling with sequence length.
Solution Approach 2:
The system pre-computes and stores the Q and K matrices in SRAM before the attention computation begins. During the pipeline execution, these pre-loaded matrices are rapidly accessed without requiring repeated computation or external memory access, significantly improving throughput and reducing the effective computational complexity during the attention operation.
2Reliability
If division operation in softmax computation is performed early, then numerical stability is improved, but computational time and energy consumption increase
Solution Approach 1:
The system performs the division operation in the softmax stage as a preliminary action before the final PV multiplication. By computing the softmax normalization coefficients early and storing them for reuse during the value transformation phase, the system ensures numerical stability while avoiding repeated division operations, thus reducing overall computation time and energy consumption.
3Use of energy by moving object
If analog AI accelerators are used for matrix-matrix multiplication, then energy efficiency is improved for weight-activation multiplication, but performance degrades when multiplying two activation matrices
Solution Approach 1:
The system segments the self-attention computation into three distinct stages, each optimized for its specific computational requirements. The QK and PV stages involve weight-activation multiplications that leverage the energy-efficient analog matrix multipliers, while the softmax stage uses digital processing for the division operations. This segmentation allows the system to maintain high energy efficiency while achieving acceptable performance for activation matrix multiplications.
Data Source
AI summary
A compute engine is configured to perform self-attention computations by delaying performance of a division operation of a softmax computation, the performance including iteratively computing a first matrix multiplication of a given row vector of a first matrix and each column vector of a second matrix while determining a first scalar element representing a maximum value of the iterative first matrix multiplications; iteratively subtracting a corresponding determined first scaler element from a result of each computed first matrix multiplication and computing an elementwise exponential function based on a result of the subtraction operation to generate a plurality of elements of a given row vector of a fourth matrix; iteratively computing a second matrix multiplication of a given row vector of the fourth matrix and each column vector of a third matrix while summing the given row vectors of the fourth matrix; and computing a row vector of an output matrix.