Voice Adaptation Using Siamese Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine translation systems struggle to adapt computer-generated voices to resemble human speakers, leading to synthesized speech that may not be easily understandable, as they fail to accurately replicate pitch characteristics and emphasis signals across languages.
Innovation Solution
A system that uses a Siamese neural network model to select and modify computer-generated voices based on human speaker characteristics, such as pitch and emphasis, to generate synthesized speech that closely mimics the human voice, thereby improving understanding and reducing the need for additional processing turns.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a speech synthesis model processes computer generated voices to generate synthesized speech, then the translation application can provide synthesized speech output, but the synthesized speech may not be easily understandable because the computer generated voice does not accurately replicate human pitch characteristics
Solution Approach 1:
The system modifies the computer generated voice by adjusting pitch characteristics and other audio parameters to match the detected human speaker's voice. The pitch adjustment module transforms the synthesized speech to resemble the original speaker's pitch pattern, making the speech more intelligible and natural while maintaining the translation functionality.
2Reliability
If the system uses a Siamese neural network model to select and modify computer-generated voices, then the synthesized speech closely mimics the human voice, but the device complexity increases
Solution Approach 1:
The Siamese neural network creates a simplified representation (embedding) of the human speaker's voice characteristics and copies this representation to guide the selection and modification of computer generated voices. This allows the complex task of voice adaptation to be performed through a structured process of comparison, selection, and parameter adjustment rather than requiring entirely new complex models.
3Productivity
If the system automatically adapts the computer generated voice to match the user's voice, then additional processing turns are reduced, but the computational resources required for voice analysis and modification increase
Solution Approach 1:
The system performs voice analysis and creates voice representations in advance using the Siamese neural network. By pre-processing the human speaker's voice characteristics and storing them as templates, the system can quickly match and adapt computer generated voices without requiring extensive real-time computational resources for each conversation turn.
Data Source
AI summary
A computer generated voice can automatically be adapted to be similar to a user's voice. Various implementations include processing audio data capturing a first language spoken utterance to identify one or more pitch characteristics. For example, the one or more pitch characteristics can include an estimated frequency range of the given user's voice. Additionally or alternatively, the system can process the audio data capturing the first language spoken utterance and a set of candidate computer generated voices using a computer generated voice selection model to select a candidate computer generated voice. Various implementations can include automatically modifying the selected candidate computer generated voice based on the one or more pitch characteristics to change the frequency range of the modified computer generated voice based on the user's voice.