Language Model Monotonic Alignment Training for Speech Synthesis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Language model-based text-to-speech synthesis systems struggle with robustness when handling input text with multiple occurrences of the same text tokens, leading to issues like repeating words, missing words, and misaligned speech due to the auto-regressive nature of these models and the lack of constrained alignment between text and speech.
Innovation Solution
Training language models using attention priors and connectionist temporal classification (CTC) losses to enforce monotonic alignment between text and speech tokens, improving the model's ability to align inputs and outputs sequentially.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If language models are trained using conventional implicit alignment methods, then the model can generate speech outputs, but the alignment between text tokens and speech tokens becomes unreliable leading to repeating words, missing words, and misaligned speech
Solution Approach 1:
The patent changes the alignment parameter constraints by introducing monotonicity requirements. Specifically, it enforces that attention scores must be monotonically non-increasing with respect to text token positions and monotonically non-decreasing with respect to speech token positions. This parameter transformation converts the unreliable implicit alignment into a constrained monotonic alignment that prevents repeating and missing words while maintaining speech fluency.
Solution Approach 2:
The patent introduces attention scores as an intermediary mechanism to mediate the alignment between text tokens and speech tokens. By computing and constraining attention scores that represent the relationship between text and speech positions, the model gains a controllable intermediate representation that enforces proper alignment. This intermediary attention mechanism allows the model to track which speech token corresponds to which text token, preventing alignment errors.
2Productivity
If language models use auto-regressive generation, then the model can generate sequential speech tokens, but the same text tokens may cause repeated speech tokens due to the sequential nature of generation
Solution Approach 1:
The patent modifies the generation parameters by imposing monotonic constraints on attention scores during auto-regressive generation. The constraint ensures that once a text token's corresponding speech token position is identified through attention, subsequent speech tokens cannot attend to the same or previous text tokens. This parameter transformation maintains the sequential generation capability while preventing repetition by enforcing strictly increasing attention position mappings.
3Ease of manufacture
If language models are trained without monotonic alignment constraints, then the training process is simpler and faster, but the model produces speech with alignment errors and hallucinations
Solution Approach 1:
The patent introduces monotonicity constraints as additional parameter conditions during training. By transforming the alignment parameter space to enforce monotonic relationships between text and speech token positions, the training process gains guidance that improves alignment precision. The constraint is implemented through modified loss functions that penalize non-monotonic attention patterns, providing clear optimization direction while maintaining training feasibility.
Data Source
AI summary
In various examples, learning monotonic alignment for language models in AI systems and applications is described herein. Systems and methods are disclosed that train one or more language models—such as LLMs or VLMs—using one or more techniques that improve the ability of the language model(s) to align inputs (e.g., text tokens) with outputs (e.g., speech tokens). For instance, to learn a stricter alignment and improve robustness of the language model(s), the training may encourage monotonic cross-attention scores using one or more attention priors and/or using one or more connectionist temporal classification (CTC) losses when updating the language model(s). For instance, the attention prior(s) may initialize the cross-attention scores to a monotonic heuristic while the CTC loss(es) may ensure the learned alignment attends over one or more text tokens (e.g., all text tokens) sequentially.