Conversational LLM Speech Pipeline With Split Response Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) are unsuitable for implementation on user devices due to high computational requirements and introduce unacceptable latency and jitter when implemented on remote servers, impeding natural conversational experiences.
Innovation Solution
A two-tiered model approach where a small, low-latency first model executes on the user device to generate an initial response portion, while a larger LLM on a remote system generates the remaining response, masking latency by starting the audible output shortly after the utterance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a large language model is implemented on a remote server to provide conversational responses, then the model can generate comprehensive responses, but latency and jitter increase making natural conversation difficult
Solution Approach 1:
The response generation process is divided into two segments: a first model generates an initial response portion locally on the user device with low latency, while a second large language model generates the remaining portion on a remote server. This segmentation allows the system to provide both low-latency initial responses and comprehensive final responses, resolving the contradiction between response quality and latency.
2Loss of time
If a large language model is implemented on a user device to provide immediate responses, then latency is reduced, but computational requirements become too high for the device
Solution Approach 1:
The system segments the computational load by deploying a small first model on the user device for low-latency initial response generation, and a larger second model on a remote server for comprehensive response generation. This segmentation allows the user device to maintain low latency without bearing the full computational burden of a large language model.
Solution Approach 2:
The first model acts as an intermediary between the user device and the second large language model. It generates an initial response portion that can be immediately output, masking the latency of the subsequent processing by the second model on the remote server. This intermediary approach reduces the apparent latency while maintaining device simplicity.
3Reliability
If the full response is generated before output to ensure accuracy, then response quality is maintained, but latency increases making conversation feel unnatural
Solution Approach 1:
The first model performs preliminary action by generating an initial response portion before the second model completes the full response. This initial portion is output immediately to mask latency, while the second model subsequently generates and appends the remaining portion. This preliminary action maintains response quality through the second model's comprehensive processing while reducing perceived latency through early output.
Data Source
AI summary
A method includes receiving a transcription of an utterance, processing, using a first model, the transcription to generate a first text segment that represents an initial portion of a response to the utterance, processing, using a TTS system, the first text segment to generate a first synthesized speech representation, and providing, for audible output, the first synthesized speech representation. The method also includes providing, to a second model different from the first model, the transcription and the first text segment, the second model comprising an LLM configured to process the transcription and the first text segment to generate a second text segment that represents a remaining portion of the response to the utterance. The method further includes obtaining a second synthesized speech representation generated from the second text segment, and providing, for audible output by the user device, the second synthesized speech representation.