Phone-Label Guided Text-to-Speech Training for Distinct Sounds
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speech synthesis systems struggle to generate distinct speech sounds, particularly fricative consonants and vowels, despite their spectral closeness in the signal space, leading to unintelligible synthetic speech due to the reliance on loss functions that do not account for perceptual distinctions.
Innovation Solution
A text-to-speech model is trained using a phone label mapping network and a cross-entropy loss term to predict phone labels for each frame of speech representation, aligning predicted audio features with frame-aligned reference phone labels to enhance the intelligibility of synthesized speech.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional loss functions are used to train speech synthesis systems, then the spectral distance between hypothesized and reference speech signals is minimized, but the perceptual distinctness of speech sounds deteriorates
Solution Approach 1:
The patent introduces a phone label mapping network as an intermediary component that maps speech representations to phone labels. This intermediary translates the continuous spectral space into discrete perceptual categories, allowing the system to optimize for both spectral accuracy and perceptual distinctness simultaneously. The phone label mapping network acts as a bridge between the spectral domain and the perceptual domain.
Solution Approach 2:
The patent changes the optimization parameter from purely spectral distance to a combined objective that includes phone label prediction accuracy. By incorporating cross-entropy loss for phone label prediction alongside spectral loss, the training objective transitions from minimizing spectral distance to maximizing both spectral fidelity and perceptual correctness. This parameter change resolves the contradiction by making the system accountable for perceptual distinctness.
2Manufacturing precision
If speech synthesis systems focus on minimizing spectral distance, then the overall speech quality is improved, but the distinctness of specific speech sounds (e.g., fricative consonants) deteriorates
Solution Approach 1:
The patent applies local quality by making different parts of the speech representation serve different functions. The phone label mapping network focuses on specific critical speech sounds (like fricative consonants) that are difficult to distinguish, applying specialized processing to these local regions. This allows the system to maintain high overall quality while ensuring distinctness in locally critical areas.
Solution Approach 2:
The patent substitutes the traditional mechanical approach of directly minimizing spectral distance with a neural network-based phone label mapping system. Instead of relying solely on spectral optimization, the system uses learned mappings from speech representations to phone labels, allowing the network to capture complex perceptual relationships that simple spectral distance metrics cannot capture.
3Reliability
If frame-aligned phone labels are used during training, then the distinctness of speech sounds is improved, but the computational complexity of the training process increases
Solution Approach 1:
The patent segments the speech signal into frame-aligned units and assigns phone labels to each frame independently. This segmentation allows the phone label mapping network to process each frame separately, making the training process more manageable despite the increased complexity. The segmentation approach enables parallel processing and simplifies the optimization landscape.
Solution Approach 2:
The patent implements feedback by using the phone label mapping network to provide additional training signals during the training process. The predicted phone labels are compared with ground truth labels, and the resulting loss is fed back to update the speech synthesis system. This feedback mechanism guides the system toward generating more distinct speech sounds while managing training complexity through efficient loss computation.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method (800) of training a text-to- speech (ITS ) model (108) includes obtaining training data (150) including reference input text (104) that includes a sequence of characters, a sequence of reference audio features (402) representative of the sequence of characters, and a sequence of reference phone labels (502) representative of distinct speech sounds of the reference audio features. For each of a plurality of time steps, the method includes generating a corresponding predicted audio feature (120) based on a respective portion of the reference input text for the time step and generating, using a phone label mapping network (510), a corresponding predicted phone label (520) associated with the predicted audio feature. The method also includes aligning the predicted phone label with the reference phone label to determine a corresponding predicted phone label loss (622) and updating the ITS model based on the corresponding predicted phone label loss.