Task-Oriented Dialogue Model Training with Masked Language Loss
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing task-oriented dialogue language models trained on general text corpora, such as Wikipedia or social media, perform subpar due to linguistic differences between written text and real-world conversational patterns, leading to inefficient task-oriented dialogue systems.
Innovation Solution
Pre-training task-oriented dialogue language models using English-based task-oriented datasets, specifically human-human and multi-turn corpora, with masked language loss and contrastive learning techniques to improve performance on downstream tasks like intent detection, dialogue state tracking, and response selection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If existing task-oriented dialogue language models are trained on general text corpora (Wikipedia, social media), then the training data availability is high, but the performance on task-oriented dialogue tasks is subpar
Solution Approach 1:
The patent applies local quality by using different training data sources for different purposes: general text corpora (Wikipedia, books) for language understanding foundations, and task-oriented dialogue corpora (DSTC2, MWOZ, Taskmaster) for task-specific dialogue skills. This targeted approach ensures each data source contributes its strengths to specific aspects of model performance.
Solution Approach 2:
The training process is segmented into multiple stages: pre-training on general text corpora for language understanding, then fine-tuning on task-oriented dialogue corpora for specific dialogue skills. This segmentation allows the model to progressively acquire different capabilities rather than attempting to learn everything from a single data source.
2Ease of operation
If models are trained on chit-chat corpora from social media, then the conversational fluency may be improved, but the task-oriented performance deteriorates due to noisy and task-less nature
Solution Approach 1:
The patent assigns different functional roles to different corpora: chit-chat corpora (Twitter, Reddit) are used specifically for improving conversational fluency and natural language understanding, while task-oriented corpora (DSTC2, MWOZ, Taskmaster) are used for teaching specific dialogue skills and task completion. This localized use of each corpus's strengths resolves the contradiction.
Solution Approach 2:
The model undergoes preliminary training on general text and chit-chat corpora to establish strong language understanding and conversational fluency before being fine-tuned on task-oriented corpora. This preliminary action ensures the model has solid foundational skills before acquiring task-specific capabilities.
3Measurement precision
If massive scale general text corpora are used for training, then the language understanding capability is improved, but the alignment with real-world conversational patterns deteriorates
Solution Approach 1:
The patent uses different corpora for different linguistic aspects: massive general text corpora (Wikipedia, books) for deep language understanding and semantic knowledge, while task-oriented dialogue corpora (DSTC2, MWOZ, Taskmaster) for authentic conversational patterns and task-oriented language use. This targeted assignment ensures both language understanding and conversational alignment are optimized.
Data Source
AI summary
Embodiments described herein provide methods and systems for training task-oriented dialogue (TOD) language models. In some embodiments, a TOD language model may receive a TOD dataset including a plurality of dialogues and a model input sequence may be generated from the dialogues using a first token prefixed to each user utterance and a second token prefixed to each system response of the dialogues. In some embodiments, the first token or the second token may be randomly replaced with a mask token to generate a masked training sequence and a masked language modeling (MLM) loss may be computed using the masked training sequence. In some embodiments, the TOD language model may be updated based on the MLM loss.