Acoustic Semantic Language Model for Speech Dataset Augmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speech processing systems, particularly Text-to-Speech (TTS) models, face challenges in achieving high speech quality and intelligibility due to the need for large and diverse training datasets.
Innovation Solution
The system employs an acoustic/semantic language model (ASLM) to generate an augmented speech dataset by predicting continuations of speech based on a target dataset, using encoded speech units that include semantic and acoustic information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If large and diverse training datasets are used to train TTS models, then speech quality and intelligibility are improved, but data collection time and resource requirements increase
Solution Approach 1:
The system performs preliminary encoding of speech data into semantic and acoustic features in advance. The ASLM model pre-learns speech patterns and continuations from the target voice dataset, preparing augmented speech data that can be directly used for TTS training without requiring extensive additional data collection
Solution Approach 2:
The system creates synthetic copies of speech data by generating augmented speech samples that mimic the characteristics of the target voice. The ASLM model generates continuation speech units that are synthesized to match the acoustic and semantic properties of the original voice, effectively copying voice characteristics without requiring additional real speech recordings
2Manufacturing precision
If large and diverse training datasets are collected, then TTS model performance is improved, but storage requirements and data processing complexity increase
Solution Approach 1:
The system extracts only the essential semantic and acoustic features from speech data using encoder models. Instead of storing and processing complete speech waveforms, the system works with compressed feature representations (encoded speech units) that capture the critical information needed for TTS training, significantly reducing data volume
Solution Approach 2:
The system transforms speech data from its original waveform form into encoded feature space with specific semantic and acoustic parameters. This parameter transformation allows the ASLM model to generate augmentation data in the same feature space, maintaining consistency while reducing the dimensional complexity and storage requirements compared to raw audio data
3Loss of time
If limited voice datasets are used for training, then data collection resources are reduced, but speech quality and intelligibility deteriorate
Solution Approach 1:
The ASLM model acts as an intermediary that bridges the gap between limited training data and high-quality speech output. It generates intermediate augmented speech data by predicting continuation speech units based on the input features, effectively mediating between the constrained input data and the requirement for diverse training examples
Solution Approach 2:
The system applies parameter changes by transforming the limited input speech into encoded features and then generating augmented continuations with modified semantic and acoustic parameters. This allows the model to create diverse training samples from limited source material, maintaining speech intelligibility without requiring extensive data collection
Data Source
AI summary
A target voice dataset may be augmented using speech prediction. Encoder and decoder models may be trained to encode audio data into encoded speech data and convert it back to audio. The encoded units may include semantic information (e.g., phonemes and/or words) as well as feature data indicating prosody, timbre, speaker identity, speech style, emotion, etc. of speech. An acoustic/semantic language model (ASLM) may be configured to predict encoded speech data in a manner analogous to a language model predicting words; for example, based on preceding encoded speech data. The models may be used to generate synthesized speech samples having voice characteristics (e.g., feature data) similar to those of the target voice dataset. The augmented dataset may be used to train a text-to-speech (TTS) model to reproduce the target voice characteristics, and may improve performance of the TTS model over training with only the original target voice dataset.