Speech Normalizer for Textless Speech-to-Speech Translation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speech-to-speech translation systems face challenges in translating speech from one language to another without the need for intermediate text transcription, particularly when dealing with real-world data that includes variations due to accents and recording conditions from multiple speakers.
Innovation Solution
A self-supervised unit-based speech normalization technique is used to finetune a pre-trained speech encoder with paired audio from multiple speakers and a reference speaker, reducing variations while maintaining the lexical content, allowing for effective training of a textless speech-to-speech translation model using real-world data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If synthetic speech data is used to train textless speech-to-speech translation models, then the training data is clean and consistent, but it does not reflect real-world scenarios with background noise and poor data collection conditions
Solution Approach 1:
The patent applies speech normalization as a pre-processing step before training the speech-to-speech translation model. By normalizing the real-world speech data in advance to match the reference speaker's characteristics, the system prepares the data to reduce speaker variations beforehand, enabling the model to learn from real-world data effectively without being exposed to the inconsistencies during training
2Adaptability or versatility
If real-world speech data from multiple speakers is used for training, then the system can handle diverse accents and recording conditions, but the utterances of the same content by different people sound different due to accent variations
Solution Approach 1:
The patent changes the acoustic parameters of speech signals from multiple speakers to match the reference speaker's characteristics. The speech normalizer modifies parameters such as pitch, timbre, and spectral features to reduce speaker-specific variations while preserving the lexical content, thereby achieving consistency across different speakers
3Measurement precision
If a speech normalizer is trained to remove speaker variations, then speech signals from different speakers look roughly the same, but the system requires additional processing steps and computational resources
Solution Approach 1:
The patent introduces a speech normalizer as an intermediary component between the input speech signal and the speech-to-speech translation model. This normalizer acts as a mediator that transforms multi-speaker speech into a unified representation matching the reference speaker, simplifying the input for the main translation model while isolating the complexity of speaker normalization in a dedicated module
Data Source
AI summary
In one embodiment, a method includes accessing a first utterance of a content by a first speaker, generating first discrete speech units from the first utterance based on a speech-learning model, wherein each of the first discrete speech units is associated with a speech cluster, accessing second utterances of the content by second speakers different from the first speaker, and training a speech normalizer by processing each of the second utterances using the speech normalizer to generate second discrete speech units and updating the speech normalizer by using the first discrete speech units as an optimization target for the second discrete speech units associated with each of the second utterances.