Dialogue Model Distillation for Low-Latency Response Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing generative-based dialogue models suffer from high latency in real-time conversations due to auto-regressive decoding, while retrieval-based models lack conversational ability and flexibility due to reliance on pre-defined response sets.
Innovation Solution
A training method called generative-to-retrieval distillation (G2R) is employed, which augments the dialogue dataset with responses generated by a large-scale generative model and transfers response quality scores from the generative model to the retrieval model, leveraging the knowledge of both models to improve efficiency and conversational ability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a generative-based dialogue model is used to generate responses, then response quality and conversational ability are improved, but response latency increases significantly
Solution Approach 1:
The system pre-generates and stores a large corpus of dialogues in a database before actual conversation occurs. During real-time interaction, the system retrieves relevant pre-generated dialogues using retrieval-based methods rather than generating responses on-demand, thus reducing latency while maintaining quality through the pre-computed knowledge base.
Solution Approach 2:
The patent introduces an intermediary retrieval mechanism that bridges the gap between user queries and appropriate responses. Instead of directly generating responses (which causes high latency) or simply retrieving from fixed sets (which limits quality), the system uses retrieval-based methods to access pre-generated dialogues, effectively mediating between speed and quality requirements.
2Loss of time
If a retrieval-based dialogue model is used to return responses from a pre-defined response set, then response latency is reduced, but response quality and conversational ability deteriorate
Solution Approach 1:
The system changes the parameter of response storage from fixed pre-defined sets to a dynamic database containing diverse pre-generated dialogues. This allows the retrieval-based model to access a richer variety of responses while maintaining fast retrieval speeds, thus improving response quality without sacrificing latency performance.
Solution Approach 2:
The patent expands the response space from a limited pre-defined set to a comprehensive database of pre-generated dialogues. This dimensional expansion allows the retrieval-based model to access responses across multiple dimensions (different topics, styles, lengths) while maintaining the speed advantage of retrieval over generation.
3Quantity of substance
If a large-scale language model is used for generative responses, then knowledge richness is improved, but decoding time and computational complexity increase
Solution Approach 1:
The system performs the knowledge-intensive generation of diverse dialogues in advance using large-scale language models, storing the results in a database. During actual use, the system only performs fast retrieval operations rather than expensive generation, thus maintaining knowledge richness while significantly reducing real-time computational complexity.
Solution Approach 2:
Instead of using a single large-scale language model that requires complex decoding for each response, the system creates copies of generated dialogues and stores them in a database. During interaction, the system retrieves these pre-generated copies rather than performing new generation, reducing decoding complexity while preserving the knowledge richness captured in the stored dialogues.
Data Source
AI summary
Disclosed is a method of training a dialogue model in an electronic device, the method including selecting a first context from a first dialogue data set including at least one pair of a context and a response corresponding to the context, generating a first response corresponding to the first context through a first dialogue model, generating an augmented dialogue dataset by incorporating a pair of the first context and the first response corresponding to the first context into the first dialogue data set, and training a second dialogue model based on the augmented dialogue dataset.