Speech Translation Voice Cloning via Acoustic Feature Modification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing spoken language translation systems struggle to generate speech in a target language that accurately captures the prosody and voice characteristics of a specific speaker, often resulting in unnatural speech output due to the inability to model context and retain speaker-specific acoustic features.
Innovation Solution
The system extracts acoustic features from a speech signal in the target language spoken by a different voice and modifies them to match the characteristics of the desired speaker, using a trained text-to-speech synthesis model that incorporates these modified features to generate output speech that sounds natural and consistent with the target speaker.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If acoustic features are extracted from a speech signal in the target language spoken by a different voice, then the system can generate speech with correct language translation, but the output speech lacks naturalness and speaker-specific acoustic characteristics
Solution Approach 1:
The system segments the acoustic feature generation process into two independent components: (1) extracting acoustic features from a reference speech signal spoken by the desired speaker, and (2) modifying these features to match the target language phonetics. This segmentation allows preservation of speaker-specific characteristics while achieving accurate language translation, resolving the contradiction between maintaining acoustic fidelity and achieving linguistic accuracy.
Solution Approach 2:
The system introduces an intermediary acoustic feature modification process that acts as a bridge between the reference speech signal and the target language output. This intermediary process extracts acoustic features from the reference signal, modifies them to match target language phonetic requirements, and then uses these modified features to generate natural-sounding speech in the target language while retaining the original speaker's voice characteristics.
2Device complexity
If the system uses a text-to-speech synthesis model without modifying acoustic features, then the processing is simpler, but the generated speech does not accurately capture the desired speaker's voice characteristics
Solution Approach 1:
The system performs preliminary extraction of acoustic features from a reference speech signal before the text-to-speech synthesis process. By pre-extracting and pre-modifying the acoustic features to match both the target language and the desired speaker's characteristics, the system prepares optimized input for the TTS model, thereby achieving high voice characteristic accuracy without significantly increasing overall processing complexity.
Solution Approach 2:
The system changes key acoustic parameters (such as fundamental frequency, spectral characteristics, and temporal features) of the extracted acoustic features to match the desired speaker's voice characteristics while maintaining compatibility with the target language phonetics. This parameter modification approach enables accurate voice replication through the TTS model without requiring complex architectural changes.
Data Source
Figure 1
Figure 2(a)
Figure 2(b)
AI summary
A computer implemented speech processing method for generating translated speech comprising: receiving a first speech signal corresponding to speech spoken in a second language; generating first text data from the first speech signal, the first text data corresponding to text in the second language; generating second text data from the first text data, the second text data corresponding to text in a first language; responsive to obtaining a second speech signal corresponding to the second text spoken in the first language and in a second voice: extracting first acoustic data from the second speech signal; modifying the first acoustic data based on one or more acoustic data characteristics corresponding to a first voice; and generating an output speech signal using a text to speech synthesis model taking the second text data as input and using the modified first acoustic data, the output speech signal corresponding to the second text spoken in the first language.