Transformer Encoder Masked Self-Attention for Real-Time Sequence Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Typical sequence-to-sequence neural networks require the entire input sequence to generate output, making them unsuitable for real-time applications like machine interpretation and speech recognition, where partial input sequences cannot produce accurate and consistent partial output sequences.
Innovation Solution
A processor-implemented method using an encoder to determine self-attention weights based on preceding tokens in the input sequence, masking relationships with following tokens, and a decoder to generate output sequences, allowing for real-time partial output generation by updating context information as tokens are input.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a typical sequence-to-sequence neural network is used to determine context information for the entire input sequence, then the output sequence can be generated based on complete context information, but the network cannot generate partial output sequences based on partial input sequences, making it unsuitable for real-time applications
Solution Approach 1:
The patent divides the input sequence processing into incremental segments. Instead of requiring the entire input sequence to generate output, the encoder processes tokens incrementally, generating context information for each prefix of the input sequence. This allows the decoder to generate partial output sequences at each step, enabling real-time processing while maintaining accuracy through progressive context accumulation.
Solution Approach 2:
The patent performs preliminary encoding of each token as it arrives, maintaining encoded representations of all previously processed tokens. This preliminary action allows the model to immediately begin processing new tokens without waiting for the complete input sequence, enabling real-time partial output generation while preserving the ability to incorporate future context as it becomes available.
2Loss of information
If self-attention weights are calculated considering all tokens in the input sequence, then complete context information is captured, but the model cannot process partial input sequences incrementally for real-time applications
Solution Approach 1:
The patent implements dynamic self-attention masking that adapts based on the current processing step. The attention mask dynamically expands to include newly arrived tokens while maintaining visibility to all previous tokens. This dynamic adjustment allows the model to capture complete context information for each prefix of the input sequence, enabling incremental processing without losing contextual completeness.
Solution Approach 2:
The model performs preliminary encoding and attention calculation for each token as it arrives, rather than waiting for the complete sequence. This preliminary action with dynamic masking ensures that context information is captured incrementally and accurately at each step, reducing processing time delay while maintaining information completeness.
3Reliability
If the entire input sequence is required to generate any portion of the output sequence, then accurate output can be produced, but the system cannot provide real-time results for streaming applications
Solution Approach 1:
The patent segments the output generation process to occur incrementally as input tokens are processed. The decoder generates portions of the output sequence corresponding to each processed prefix of the input sequence. This segmentation allows real-time streaming capability while maintaining reliability, as each partial output is generated based on complete processing of the corresponding input prefix available at that moment.
Solution Approach 2:
The patent maintains continuous processing by immediately encoding new tokens as they arrive and continuously updating context information. The decoder continuously generates output tokens based on the accumulated context, providing uninterrupted real-time streaming output. This continuity ensures both real-time capability and reliability through consistent, ongoing processing without idle waiting periods.
Data Source
AI summary
A processor-implemented method includes: using an encoder, determining, for each of a plurality of tokens included in an input sequence, a self-attention weight based on a token and one or more tokens that precede the token in the input sequence; using the encoder, determining context information corresponding to the input sequence based on the determined self-attention weights; and using a decoder, determining an output sequence corresponding to the input sequence based on the determined context information.