Unified Language Model Pre-Training With Chunked Attention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing NLP models face inefficiencies in performing both natural language generation (NLG) and understanding (NLU) tasks, particularly when processing large-scale and long-form data, due to cumbersome attention processing.
Innovation Solution
A unified language foundation model is trained using a multi-step process that integrates a generative language model with a replaced token detection (RTD)-like discriminative objective and corrupted span reconstruction, applying disentangled attention to discrete data chunks for improved efficiency and effectiveness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional NLP models apply attention to all data tokens to improve contextual accuracy, then NLU task performance is improved, but processing efficiency deteriorates significantly when handling long-form data
Solution Approach 1:
The patent segments the input sequence into discrete chunks of tokens, applying attention mechanisms only within each chunk rather than across the entire sequence. This segmentation reduces the computational complexity from O(n²) to O(m²×k) where n is total tokens, m is tokens per chunk, and k is number of chunks, thereby improving processing efficiency while maintaining contextual accuracy through sufficient chunk size selection
2Measurement precision
If huge generative foundation models are trained to perform NLG tasks, then generation quality is improved, but training time and computational resources increase immensely
Solution Approach 1:
The patent merges NLU and NLG training objectives into a unified pre-training framework. By combining discriminative tasks (masked language modeling, next sentence prediction) with generative tasks (text generation, summarization) in a single model architecture and training process, the system achieves both NLU and NLG capabilities without requiring separate huge generative models, thereby reducing training time and computational resources
Solution Approach 2:
The patent creates a universal pre-trained model that performs multiple functions including NLU tasks (classification, extraction), NLG tasks (generation, summarization), and language modeling. This multi-functional model eliminates the need for specialized huge generative models for each task, reducing overall training requirements while maintaining high generation quality
3Adaptability or versatility
If a unified foundation model is trained to perform both NLU and NLG tasks, then model versatility is improved, but the amount of training data and parameters required increases immensely
Solution Approach 1:
The patent merges multiple task objectives (masked language modeling, next sentence prediction, text generation) into a single unified training framework that shares parameters across tasks. This parameter sharing and joint training approach reduces the total number of parameters compared to training separate specialized models, while achieving versatility across NLU and NLG tasks through multi-objective optimization
Data Source
AI summary
Systems and methods are provided for training and using a novel unified language foundation model. An encoder-decoder natural language model is obtained and various training data is obtained and used for training. The training process integrates a combination of replaced token detection, corrupted span reconstruction, and disentangled attention methodologies to produce a unified encoder-decoder model. The trained model is trained for performing both natural language understanding (NLU) tasks and natural language generation (NLG) tasks. Attention applied to the model is applied discretely to segmented chunks of encoded data during processing to improve the efficiency of applying attention by the model.