Butterfly Processor Neural Network Accelerator
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing hardware accelerators for machine learning computations, particularly attention-based neural networks, face challenges in scalability and efficiency due to their focus on optimizing either feed-forward networks or attention mechanisms separately, leading to high power consumption and limited parallelism, and require complex dynamic controllers for sparsity exploitation, which increases hardware overhead.
Innovation Solution
The proposed solution involves a hardware accelerator with a butterfly processor that generates query, key, and value matrices through Fast Fourier Transform and butterfly linear transforms, allowing for efficient matrix multiplications and softmax operations, and an attention processor for handling attention layers without butterfly sparsity, enabling scalable end-to-end acceleration of attention-based neural networks with static sparsity patterns.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If dynamic sparsity detection and pruning is used to optimize attention mechanism, then computation efficiency is improved, but device complexity increases due to required hardware controllers
Solution Approach 1:
The patent pre-determines sparsity patterns during the training phase and stores them for use during inference. This eliminates the need for complex runtime controllers that would otherwise be required to dynamically detect and prune redundant computations, while still achieving significant computation efficiency improvements.
2Speed
If high degree of parallelism is implemented in attention-based NNs, then processing speed is improved, but power consumption increases significantly
Solution Approach 1:
The patent extracts and eliminates redundant computations from the attention mechanism by applying pre-determined sparsity patterns. This reduces the amount of parallel computation required, thereby lowering power consumption while maintaining acceptable processing speed through selective computation of only necessary attention weights.
3Productivity
If existing hardware accelerators optimize only FFN or attention mechanism separately, then specific operation efficiency is improved, but scalability to end-to-end models is limited
Solution Approach 1:
The patent designs a unified hardware accelerator that can handle both FFN and attention mechanism operations within a single architecture. The system uses configurable compute units that can be dynamically allocated to different operation types, enabling the accelerator to scale effectively to end-to-end models while maintaining high efficiency for both operation types.
Data Source
AI summary
Broadly speaking, the present disclosure generally relate to an electronic device for accelerating machine learning, ML, model computations is provided. The electronic device comprises: a first processor configured to: generate a query matrix, a key matrix, and a value matrix by performing Fast Fourier Transform (FFT) and butterfly linear transform on at least one input matrix, and a second processor configured to: perform a first matrix multiplication between the query matrix and the key matrix, perform a softmax operation on the result of the first matrix multiplication, and perform a second matrix multiplication between the result of the softmax operation and the value matrix.