Transformer Language Models Disentangled Attention Multi-Step Decoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current transformer-based language models, such as BERT, fail to accurately incorporate sequential dependencies between tokens, leading to loss of sequencing information and reduced prediction accuracy due to entanglement of position and content information during processing.
Innovation Solution
The implementation of a computing system that disentangles position bias information from content information within self-attention matrices using a summation of attention score components with disentangled projection matrices, and employs multi-step decoding to enhance pre-training convergence and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If position and content information are entangled during processing in transformer models, then the model structure is simpler, but sequencing information is lost and prediction accuracy deteriorates
Solution Approach 1:
The patent segments position information and content information into separate processing streams within the transformer model. Positional encoding is applied independently from content embeddings, allowing the model to preserve sequencing information while maintaining structural organization. This segmentation enables accurate tracking of token positions without complicating the overall architecture excessively.
Solution Approach 2:
The patent extracts position information from the entangled position-content representation and processes it separately through dedicated positional encoding mechanisms. By taking out position information for separate handling, the model preserves sequencing dependencies while content information processes semantic meaning, thereby improving prediction accuracy without proportionally increasing complexity.
2Reliability
If conventional self-attention mechanisms are used, then the implementation is simpler, but sequential dependency tracking between tokens deteriorates
Solution Approach 1:
The patent applies local quality by enhancing the self-attention mechanism specifically for positional information while keeping content processing relatively standard. Positional encodings are injected at specific locations in the network architecture, providing localized improvement in sequential dependency tracking without requiring complete redesign of the entire attention mechanism.
Solution Approach 2:
The patent introduces positional encoding as an intermediary element that mediates between the input tokens and the self-attention mechanism. This intermediary carries sequencing information through the network layers, enabling the attention mechanism to track sequential dependencies more reliably without fundamentally changing its core operation.
3Measurement precision
If position information is not separately encoded, then the processing is faster, but relevance of position information in predictions deteriorates
Solution Approach 1:
The patent applies preliminary action by pre-computing and injecting positional encodings into the input representations before they enter the main processing pipeline. This advance preparation ensures position information is readily available and relevant during prediction without requiring additional computational steps during the main processing phase, thereby maintaining processing speed.
Solution Approach 2:
The patent merges position information with content information at strategic points in the network architecture where it enhances prediction relevance. By combining these information streams at appropriate locations rather than keeping them completely separate throughout, the model maintains processing efficiency while ensuring position information remains relevant for final predictions.
Data Source
AI summary
Systems and methods are provided for facilitating the building and use of natural language understanding models. The systems and methods identify a plurality of tokens and use them to generate one or more pre-trained natural language models using a transformer. The transformer disentangles the content embedding and positional embedding in the computation of its attention matrix. Systems and methods are also provided to facilitate self-training of the pre-trained natural language model by utilizing multi-step decoding to better reconstruct masked tokens and improve pre-training convergence.