Generative-to-Retrieval Distillation for Low-Latency Dialogue Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing generative-based dialogue models suffer from high latency in real-time conversations due to auto-regressive decoding, while retrieval-based models lack conversational ability and flexibility due to reliance on pre-defined response sets.
Innovation Solution
A training method called generative-to-retrieval distillation (G2R) is employed, which augments the dialogue dataset with responses generated by a large-scale generative model and transfers response quality scores from the generative model to the retrieval model, leveraging the knowledge of both models to improve efficiency and quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a generative-based dialogue model is used to generate responses in real-time conversations, then the conversational ability and flexibility are improved, but the response latency increases significantly due to auto-regressive decoding
Solution Approach 1:
The patent pre-generates multiple candidate responses for each dialogue context using a generative model during the training phase, storing them in a response set. During inference, the retrieval model directly selects from these pre-generated responses without performing auto-regressive decoding, thus eliminating the time-consuming generation process while maintaining access to high-quality conversational responses
Solution Approach 2:
The patent creates a distilled retrieval model that copies the conversational knowledge and response quality of the large-scale generative model into a smaller, faster model. The retrieval model learns to replicate the generative model's response selection behavior by training on pairs of contexts and the generative model's chosen responses, enabling fast inference with generative-level quality
2Loss of time
If a retrieval-based dialogue model is used with a pre-defined response set, then the response latency is reduced, but the conversational ability deteriorates due to reliance on fixed responses
Solution Approach 1:
The patent pre-generates a comprehensive response set using a large-scale generative model, where multiple high-quality candidate responses are created for each possible dialogue context. This pre-computation phase captures the generative model's conversational abilities, allowing the retrieval model to access diverse and context-appropriate responses without real-time generation
Solution Approach 2:
The retrieval model is trained to copy the response selection behavior of the generative model by learning from training data where the generative model has already selected optimal responses. The student retrieval model replicates the teacher generative model's knowledge about which responses are most appropriate for given contexts, achieving comparable conversational ability with faster inference
3Manufacturing precision
If a large-scale language model is used as a generative dialogue model, then the response quality is improved based on rich knowledge, but the computational complexity and inference time increase
Solution Approach 1:
The patent extracts the essential conversational knowledge and response selection capabilities from the large-scale generative model, separating these from the heavy computational infrastructure. The distilled retrieval model captures only the necessary knowledge for response selection, discarding the redundant computational complexity of the original generative model while preserving response quality
Solution Approach 2:
The patent creates a compact copy of the generative model's knowledge through knowledge distillation. The student retrieval model learns from the teacher generative model's training data and selected responses, copying the essential patterns and knowledge about appropriate responses without replicating the large model's architecture or computational requirements
Data Source
AI summary
Disclosed is a method of training a dialogue model in an electronic device, the method including selecting a first context from a first dialogue data set including at least one pair of a context and a response corresponding to the context, generating a first response corresponding to the first context through a first dialogue model, generating an augmented dialogue dataset by incorporating a pair of the first context and the first response corresponding to the first context into the first dialogue data set, and training a second dialogue model based on the augmented dialogue dataset.