Voice Conversion Using Mel-Spectrogram and Variational Autoencoder
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing voice conversion systems struggle to accurately extract and convert the acoustic features of whispering voices, often ignoring language information and failing to generate natural pitch and voice power, especially in noisy environments.
Innovation Solution
A signal analysis system that uses a conversion network trained with mel-spectrograms to convert the acoustic features of whispering voices into those of normal voices, employing a classifier variational autoencoder and a neural vocoder to generate high-quality target acoustic signals.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If ACVAE-VC with mel-cepstrum coefficient is used for voice conversion, then voice attribute conversion is effective, but pitch information and language information are lost in whispering voice conversion
Solution Approach 1:
The patent changes the acoustic feature parameter from mel-cepstrum coefficient to mel-spectrogram. This parameter change enables better preservation of pitch information and language information in whispering voice conversion while maintaining voice attribute conversion effectiveness. The mel-spectrogram contains more comprehensive spectral information compared to mel-cepstrum, allowing the conversion network to retain crucial linguistic and prosodic features.
2Ease of manufacture
If non-parallel voice conversion is used, then data collection is easier, but extraction accuracy of acoustic features deteriorates
Solution Approach 1:
The patent introduces a conversion network based on conditional variational autoencoder as an intermediary that processes mel-spectrogram features. This intermediary structure enables the system to achieve high acoustic feature extraction accuracy using non-parallel data by learning the mapping between source and target voice characteristics without requiring aligned parallel corpora. The auxiliary classifier further mediates to ensure accurate attribute conversion.
3Device complexity
If whispering voice is converted directly, then conversion process is simple, but clarity and intelligibility are insufficient
Solution Approach 1:
The patent applies preliminary action by pre-processing the whispering voice through mel-spectrogram extraction before conversion. This preliminary step enhances the spectral representation of the whispering voice, making pitch and language information more prominent. The conversion network then processes this enhanced representation to generate clear normal voice output, improving intelligibility without significantly increasing overall system complexity.
Data Source
AI summary
A signal analysis system includes: an acquisition unit configured to acquire a conversion network trained by using a sequence of a first mel-spectrogram in a machine learning scheme of acoustic conversion based on a classifier variational autoencoder; and a converter that converts a sequence of a second mel-spectrogram of an input acoustic signal into a sequence of a third mel-spectrogram of a target acoustic signal using the conversion network. The classifier variational autoencoder may perform learning of the conversion network using a task of complementing a missing frame in the sequence of the first mel-spectrogram. The classifier variational autoencoder may perform learning of the conversion network using a sequence of the first mel-spectrogram of the acoustic signal on which a noise signal is superimposed.