Dilated Self-Attention Reduces Neural Network Complexity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The computational complexity of self-attention mechanisms in neural networks grows quadratically with input sequence length, leading to increased processing time and storage requirements, particularly in applications like Automatic Speech Recognition (ASR), where input sequences can be long, resulting in low performance and efficiency.
Innovation Solution
The dilated self-attention mechanism combines restricted self-attention with a dilation mechanism, allowing for parallel computation of key and value frames at reduced resolution for distant information, thereby reducing computational complexity to linear growth with input sequence length, while maintaining accurate context capture.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If standard self-attention mechanism is used, then attention to all input frames is achieved, but computational complexity grows quadratically with input sequence length
Solution Approach 1:
The patent divides the input sequence into multiple segments and applies different attention mechanisms to different segments. For frames within a certain distance (k) from the current frame, standard self-attention is applied; for frames farther away, a simplified attention mechanism is used. This segmentation approach reduces the overall computational complexity while maintaining accurate context capture for both nearby and distant frames.
Solution Approach 2:
The patent applies different quality levels of attention processing to different parts of the input sequence. Nearby frames receive detailed attention processing while distant frames receive simplified processing. This local quality differentiation allows the system to maintain high accuracy for locally relevant information while reducing computational burden for distant information, resolving the contradiction between precision and complexity.
2Device complexity
If restricted self-attention mechanism is used, then computational complexity is reduced, but distant information is ignored
Solution Approach 1:
The patent segments the attention mechanism into two parts: standard self-attention for nearby frames (within distance k) and simplified attention for distant frames (beyond distance k). This ensures that distant information is not completely ignored but processed with appropriate computational resources, thus reducing information loss while controlling complexity.
Solution Approach 2:
Instead of applying full self-attention to all frames (excessive action), the patent applies partial attention - full attention to nearby frames and simplified attention to distant frames. This partial application of the computationally intensive mechanism only where most needed reduces overall complexity while preserving necessary distant context information.
3Reliability
If deeper and wider architectures are used, then model performance is improved, but processing time and storage space increase
Solution Approach 1:
The patent changes the parameters of the attention mechanism dynamically based on the distance of frames from the current frame. The attention weight calculation parameters are adjusted so that for distant frames, the attention mechanism uses simplified computations. This parameter change allows the model to maintain performance on important tasks while reducing processing time for less critical distant frame processing.
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
An artificial intelligence (AI) system is disclosed. The AI system includes a processor that processes a sequence of input frames with a neural network including a dilated self-attention module trained to compute a sequence of outputs by transforming each input frame into a corresponding query frame, a corresponding key frame, and a corresponding value frame leading to a sequence of key frames, a sequence of value frames, and a sequence of query frames of same ordering and by performing attention calculations for each query frame with respect to a combination of a portion of the sequences of key and value frames restricted based on a location of the query frame and a dilation sequence of the key frames and a dilation sequence of value frames extracted by processing different frames of the sequences of key and value frames with a predetermined extraction function. Further, the processor renders the sequence of outputs.