Singing Voice Phoneme Duration Extraction Using MIDI Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for extracting phoneme duration in singing voice synthesis are inefficient and inaccurate due to the complex characteristics of singing voices, such as breathing, vibrato, and banding, and require significant time and resources for manual annotation.
Innovation Solution
A method and system for extracting enhanced phoneme duration of singing voice phonemes using a MIDI, including a prior encoder, a posterior encoder, a flow, and a decoder, a monotonic alignment search module, and a decoder, a monotonic alignment search module, and a monotonic alignment search module, using MIDI duration information to perform monotonic alignment search on each phoneme section, thereby accurately extracting phoneme duration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If phoneme duration is extracted using MFA or VITS for singing voice synthesis, then the extraction can be automated, but the accuracy is insufficient due to complex singing characteristics such as breathing, banding, and vibrato
Solution Approach 1:
The patent segments the phoneme alignment process into two distinct stages: first using MFA for rough alignment to obtain initial phoneme boundaries, then applying a specialized dynamic time warping (DTW) algorithm for refined alignment. This segmentation allows each stage to optimize for its specific function, with the second stage correcting errors from the first stage while handling singing-specific characteristics like vibrato and breathing.
Solution Approach 2:
The patent introduces an intermediary refinement process that acts as a mediator between the automated MFA extraction and the final phoneme duration values. This intermediary stage uses singing-aware DTW with custom distance metrics that account for singing characteristics, thereby bridging the gap between automated extraction and accurate measurement.
2Measurement precision
If human annotators are used to obtain inter-phoneme boundaries for singing voice, then the accuracy of phoneme duration extraction is improved, but the time and cost required increase significantly
Solution Approach 1:
The system performs self-service by automatically refining its own phoneme alignment through the two-stage process. Instead of relying on external human annotators, the system uses its own automated MFA extraction as a starting point and then self-corrects using the singing-aware DTW algorithm, achieving high accuracy without human intervention.
Solution Approach 2:
The patent substitutes the mechanical human annotation process with an automated computational system. The singing-aware DTW algorithm replaces human listeners' perceptual judgment, using mathematical distance metrics and dynamic programming to automatically determine optimal phoneme boundaries in singing voices with complex characteristics.
3Device complexity
If standard DTW is used for phoneme alignment in singing voice synthesis, then the alignment process is simple, but the accuracy is reduced due to singing-specific characteristics
Solution Approach 1:
The patent applies local quality by making the DTW distance metric singing-specific rather than universal. Instead of using a single generic distance function, the system implements a localized distance metric that accounts for singing characteristics like vibrato frequency, breathing patterns, and pitch variations specific to each phoneme in singing contexts.
Solution Approach 2:
The system changes key parameters of the DTW algorithm to suit singing voice synthesis. This includes modifying the distance metric to incorporate pitch and spectral features, adjusting the warping constraints to accommodate singing's wider pitch ranges and longer durations, and incorporating singing-specific preprocessing steps that transform the input data into a form more suitable for DTW alignment.
Data Source
AI summary
There are provided a method and a system for extracting singing voice phoneme duration. A singing voice phoneme duration extraction system using a MIDI according to an embodiment may receive phonemes converted from a text as input, and may output a prior probability distribution, may receive acoustic features as input and may output a posterior probability distribution, may convert the probability distribution, may perform monotonic alignment search by using information on MIDI duration, and may output a waveform which is a voice digital signal, based on input reflecting a result of extracting the phoneme duration.