Singing Voice Conversion via Phoneme Alignment and Mel-Spectrogram Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional singing voice conversion systems require parallel data for training and struggle with the wide range of frequency variations and sharp changes in volume and pitch present in singing voices, limiting their ability to effectively convert singing voices without altering their content.
Innovation Solution
An unsupervised singing voice conversion method using deep neural networks that learns embedded data associated with multiple speakers during multi-speaker training, allowing for timbre conversion without parallel data, by encoding phonemes, aligning them to target acoustic frames, and recursively generating mel-spectrogram features.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional singing voice conversion systems use parallel data for training, then they can learn speaker characteristics, but they require large amounts of parallel data which limits their applicability and increases data collection complexity
Solution Approach 1:
The patent extracts and separates speaker-specific characteristics (timbre, tone) from the singing content (melody, lyrics, phonemes) by processing them through independent neural network pathways. This allows the system to convert voices without requiring parallel recordings, as it learns speaker characteristics from separate mono-speaker datasets rather than paired source-target recordings.
Solution Approach 2:
The singing voice is segmented into distinct components: phoneme sequence, pitch contour, and timbre characteristics. The system processes these separately - extracting phonemes and pitch from the input singing, then generating new timbre from a target speaker's separate training data, finally synthesizing them together. This segmentation eliminates the need for parallel data.
2Adaptability or versatility
If traditional systems handle wide frequency variations and sharp volume/pitch changes, then they can preserve singing expression, but the conversion quality deteriorates due to inability to maintain natural timbre transitions
Solution Approach 1:
The system dynamically adjusts the timbre generation process based on the instantaneous pitch and phoneme context. The neural network models are designed to handle dynamic pitch contours and volume changes by processing each time frame with context from surrounding frames, allowing natural expression preservation while maintaining stable timbre transitions through the target speaker's learned characteristics.
3Measurement precision
If the system converts singing voice timbre while preserving content, then the conversion quality improves, but the computational complexity increases due to multiple processing stages
Solution Approach 1:
The patent employs universal neural network components that perform multiple functions: the phoneme extraction network processes both speech and singing inputs, the pitch detection module handles various pitch ranges, and the timbre synthesis network adapts to different target speakers. This modular universal architecture manages complexity by reusing components across different processing stages rather than requiring separate dedicated systems for each function.
Data Source
AI summary
A method, computer program, and computer system is provided for converting a singing first singing voice associated with a first speaker to a second singing voice associated with a second speaker. A context associated with one or more phonemes corresponding to the first singing voice is encoded, and the one or more phonemes are aligned to one or more target acoustic frames based on the encoded context. One or more mel-spectrogram features are recursively generated from the aligned phonemes and target acoustic frames, and a sample corresponding to the first singing voice is converted to a sample corresponding to the second singing voice using the generated mel-spectrogram features.