Streaming Audio Generation From Real-Time LLM Text Output
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speech synthesis technologies require inputting an entire text segment before streaming synthesis can be performed, leading to significant latency and suboptimal user experience in voice interactions.
Innovation Solution
A method for generating audio using a large model that obtains characters in real time, applies a pre-trained audio generation model to generate discretized audio features for each audio unit, and synthesizes audio using a pre-trained vocoder, allowing for streaming audio synthesis with reduced latency and improved naturalness and fluency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If an entire text segment is input into a speech synthesis system before streaming synthesis can be performed, then the synthesis can be completed with full context, but the latency is significant and user experience is degraded
Solution Approach 1:
The patent segments the text generation and audio synthesis process into character-level units. Instead of waiting for complete text segments, the system processes and synthesizes audio for each character as it is generated by the large model, enabling continuous streaming output and significantly reducing overall latency.
Solution Approach 2:
The system performs preliminary preparation by having the audio generation model ready to process characters as soon as they are generated. The vocoder is pre-configured to immediately convert audio features into waveforms, eliminating idle waiting time between text generation and audio synthesis.
2Loss of time
If real-time character processing is implemented for streaming audio synthesis, then latency is reduced and user experience is improved, but the system complexity increases
Solution Approach 1:
The patent introduces an audio generation model as an intermediary component between the large language model and the vocoder. This mediator converts generated characters into audio features that the vocoder can process, enabling efficient real-time conversion without requiring complex direct integration between the language model and audio synthesis components.
3Productivity
If the audio generation model processes each character sequentially to generate discretized audio features, then streaming synthesis is achieved, but the processing overhead per character increases
Solution Approach 1:
The system changes the parameter representation from continuous text to discretized audio features. By converting characters into discrete audio unit sequences with associated feature vectors, the system enables efficient processing through the vocoder while maintaining streaming capability, as the discretized features are more computationally efficient to handle than continuous representations.
Data Source
AI summary
The present application provides a method for generating audio based on large model, an electronic device, and a storage medium, which relates to a technical field of artificial intelligence such as an audio synthesis and a large model. A specific implementation includes: obtaining a character that is generated in real time during a process of generating a text using a large model; obtaining an audio feature of each audio unit of the character sequentially by using a pre-trained audio generation model based on the character; the audio feature of the audio unit is a discretized audio feature, and the character includes audio features of a plurality of different audio units; synthesizing a corresponding audio by using a pre-trained vocoder based on the audio feature of each audio unit.