SSML Tagging for Character-Aware Audio-to-Text Conversion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional speech-to-text conversion systems fail to capture speech output characteristics of different characters, leading to monotone and robotic text-to-speech conversions that lack human-like nuances.
Innovation Solution
A system that analyzes audio corpora to identify speech output characteristics for tokens and assigns SSML tags to maintain contextual information, enabling more accurate and human-like voice synthesis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If speech-to-text conversion is performed to convert audio to text, then text output is achieved, but speech output characteristics such as pitch, emphasis, and pauses are lost
Solution Approach 1:
The patent introduces SSML (Speech Synthesis Markup Language) tags as an intermediary layer between the audio corpus and text representation. These tags act as a mediator that captures and encodes speech output characteristics (pitch, emphasis, pauses) alongside the text, enabling preservation of information that would otherwise be lost in conventional STT conversion.
Solution Approach 2:
The patent extends the traditional text representation by adding a new dimension - SSML tags that encode prosodic and paralinguistic features. This transforms the flat text output into a multi-dimensional representation that includes both linguistic content and speech characteristics, effectively adding informational layers without fundamentally changing the text-based output format.
2Ease of operation
If plain text is used for text-to-speech conversion, then simplicity is maintained, but the output sounds monotone and robotic rather than human-like
Solution Approach 1:
The patent modifies the text parameters by inserting SSML tags that specify prosodic features (pitch contours, stress patterns, pauses) and paralinguistic characteristics. These parameter changes enable the TTS system to vary speech output characteristics dynamically, transforming monotone robotic speech into more natural and expressive human-like speech while maintaining the simplicity of text-based processing.
3Loss of information
If speech-to-text conversion is applied to audio corpora with multiple characters, then text transcription is achieved, but contextual information about different characters and their speech patterns is lost
Solution Approach 1:
The patent segments the audio corpus into individual utterances or speech segments associated with specific characters, and correspondingly segments the SSML tags to mark each segment with character-specific attributes. This segmentation enables the system to capture and preserve character-specific speech patterns and contextual information while maintaining manageable data structure.
Data Source
Figure 1A
Figure 1B
Figure 2
AI summary
This disclosure describes a system that converts an audio object (e.g., an audio book, a podcast, a videoconference meeting) to text with SSML tags so that any future text-to-speech conversion enables speech synthesis to sound more human-like. The system analyzes the audio object to identify speech output characteristics for different tokens. Variations in speech output characteristics can distinguish between an utterance spoken by one character and an utterance spoken by another character. The system assigns the tokens to the characters and compares a speech output characteristic for a token to a baseline speech output characteristic associated with an identified character. Next, the system determines an amount of deviation between the speech output characteristic for the token and the baseline speech output characteristic. The system uses this deviation to determine a relative speech output characteristic value, which is to be included in an SSML tag for a token.