Digital Human Speech Segmentation Using Predicted Pause Labels
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Digital humans lack the ability to predict user questions through observation and provide responses that mimic human-like interactions, leading to reduced user engagement due to slow or non-existent physical interactions.
Innovation Solution
Implement pause-based text-to-speech processing for digital humans by using a language model to insert predicted pause labels in responses, processing tokens until a pause label or final token is detected, and transforming the response into spoken format using a text-to-speech model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If digital humans provide continuous spoken responses without pauses, then information delivery is efficient, but user engagement decreases due to lack of human-like interaction patterns
Solution Approach 1:
The response text is segmented into multiple segments based on predicted pause positions. The language model generates pause probability scores for each token, and the system divides the continuous response into discrete segments separated by pauses, mimicking natural human speech patterns while maintaining efficient information delivery.
Solution Approach 2:
The system performs preliminary action by predicting pause positions in advance using the language model before actual speech synthesis occurs. The pause labels are generated and processed ahead of time, allowing the digital human to naturally incorporate pauses without real-time computation delays.
2Speed
If digital humans process and deliver responses quickly, then responsiveness is improved, but human-like interaction qualities are lost due to absence of natural pauses
Solution Approach 1:
Pause positions are predicted in advance using the language model before speech synthesis begins. This preliminary analysis of where pauses should occur allows the system to maintain fast response delivery while naturally incorporating human-like pauses at the correct positions without real-time computation delays.
Solution Approach 2:
The system changes the parameter of speech delivery by dynamically adjusting pause duration and positioning based on the predicted pause labels. Each pause's duration can be varied independently, allowing fast overall response delivery while maintaining natural interaction qualities through controlled temporal parameters.
3Device complexity
If digital humans use traditional text-to-speech processing without pause prediction, then system complexity is reduced, but interaction naturalism deteriorates due to continuous uninterrupted speech
Solution Approach 1:
An intermediary language model component is introduced between the response generation and text-to-speech synthesis stages. This intermediary predicts pause positions by analyzing the response text and generating pause probability scores, enabling natural pauses without significantly increasing overall system complexity.
Solution Approach 2:
The language model serves itself by simultaneously generating both the response text and the pause predictions in a single pass. The same model that generates the semantic content also identifies where pauses should occur, eliminating the need for separate pause detection systems and reducing overall complexity.
Data Source
AI summary
Techniques for pause-based text-to-speech processing for digital humans are provided. One method comprises obtaining a response to be delivered by a digital human, wherein the response comprises at least one predicted pause label identifying a respective portion of the response where a human speaker is expected to pause; processing a token, comprising at least a portion of a word, from a buffer, until a predicted pause label and/or a final token indicator is detected in the token; and providing at least a portion of the given response, from the buffer, to a text-to-speech model for presentation to a user, based on the predicted pause label and/or the final token indicator in the token, wherein the digital human transforms the portion of the given response into the spoken format, using the text-to-speech model, based on the predicted pause label.