Speech Interaction Query Embedding for Streaming TTS
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current text-to-speech (TTS) systems based on large language models (LLMs) face limitations due to the inability of language encoding models to utilize previous texts and query information, leading to slow generation processes and reduced user experience.
Innovation Solution
The proposed method introduces a generative language model that utilizes query information and previous text information to enhance the representation sequence fed into a post-network, combining this with a language encoding model's output to generate answer speech in a streaming manner.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a language encoding model is used in a TTS system, then the system can process text to speech conversion, but the model cannot utilize previous texts and query information, leading to slow generation speed
Solution Approach 1:
The patent merges the language encoding model with the large language model by introducing a query embedding module. The query embedding obtains query information from the large language model and fuses it with the language encoding model's processing, enabling the system to utilize both historical context and current query information simultaneously, thereby improving generation speed without losing contextual information.
Solution Approach 2:
The query embedding module acts as an intermediary between the large language model and the language encoding model. It extracts query information from the large language model and integrates it into the language encoding process, allowing historical context to be effectively utilized without requiring the language encoding model to directly access previous texts, thus resolving the contradiction between speed and information retention.
2Loss of time
If a large language model outputs token sequence in streaming manner, then the response time is reduced, but the language encoding model cannot effectively utilize the query information and previous texts
Solution Approach 1:
The system performs preliminary action by obtaining query information from the large language model before the language encoding model processes the token sequence. The query embedding module prepares the query information in advance and integrates it with the streaming token sequence, allowing the language encoding model to effectively utilize both query information and previous texts without waiting for the complete token sequence, thus reducing response time while maintaining reliability.
Solution Approach 2:
The query embedding module implements a feedback mechanism where query information from the large language model is continuously fed back to the language encoding model during the streaming processing. This allows the language encoding model to adaptively utilize the query information and previous texts in real-time, ensuring effective information utilization while maintaining fast response times through streaming output.
3Adaptability or versatility
If the TTS system processes text sequentially without leveraging historical information, then the system complexity is low, but the user experience and interaction quality are reduced
Solution Approach 1:
The query embedding module serves multiple functions: it extracts query information from the large language model, integrates it with the language encoding model, and enables the system to utilize historical context. This multi-functional component improves user experience quality by enabling adaptive speech generation based on historical information without requiring separate complex systems for each function, thus achieving versatility with controlled complexity.
Data Source
AI summary
Embodiments of the present disclosure provide a method, a device, and a medium for speech interaction. The method includes: obtaining query information of a query text corresponding to user speech using a generative language model; obtaining, based on the query information and a token sequence, a first representation sequence of the token sequence using the generative language model, wherein the token sequence is output in a streaming manner by a large language model based on the query text; encoding the token sequence using a language encoding model to obtain a second representation sequence; and combining the first representation sequence and the second representation sequence to generate an answer speech for the user speech.