Vocal Character Swapping With CNN Spectrogram Translation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for modifying vocals require significant computation and lack flexibility, and there is a scarcity of vocal pairs with changed characteristics for training neural networks, making it difficult to achieve satisfactory results.
Innovation Solution
A pair of neural networks is trained together, with feedback loops to provide ground truth, and discriminator networks are used to determine characteristics, along with feature loss and stochastic gradient descent to improve convergence.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If conventional methods are used to modify vocals, then the modification can be achieved, but the computation required is excessive and beyond mobile device capabilities
Solution Approach 1:
The patent segments the vocal modification task into multiple components: a spectral transformation network that converts audio to spectrogram and applies style transfer, and a reconstruction network that converts the transformed spectrogram back to audio. This segmentation allows each network to specialize in a specific function, reducing the overall computational burden compared to a single comprehensive model.
Solution Approach 2:
The patent introduces an intermediary representation (spectrogram) between the input audio and output audio. The spectral transformation network transforms audio to spectrogram domain, applies style transfer in this intermediate domain, and the reconstruction network converts back to audio. This intermediary representation simplifies the transformation task and enables efficient computation on mobile devices.
2Adaptability or versatility
If conventional methods are used to modify vocals, then modification is possible, but flexibility to create various different modifications is limited
Solution Approach 1:
The spectral transformation network is designed as a universal model that can perform multiple vocal modification tasks. By training on diverse datasets with different target characteristics (different singers, styles, genders), the single model learns to generalize across multiple transformation types, providing flexibility without requiring separate models for each modification type.
Solution Approach 2:
The patent enables flexible modifications by changing the target spectral characteristics as input parameters to the trained model. By providing different target spectrograms representing different vocal characteristics, the same model can generate various modifications (different singers, styles, pitches) without retraining, achieving versatility through parameter variation.
3Manufacturing precision
If there are scarcity of vocal pairs with changed characteristics, then training data is limited, but high-quality transformation results are needed
Solution Approach 1:
The patent uses spectral copying where the target vocal style is captured as a spectrogram template. The spectral transformation network learns to copy the spectral characteristics of the target voice from these templates. By creating synthetic training pairs through spectral copying and alignment, the system generates sufficient training data without requiring extensive real-world paired vocal recordings.
Solution Approach 2:
The patent performs preliminary processing to create aligned spectral representations before actual transformation. Spectrograms are pre-processed, aligned in time and frequency domains, and normalized to create consistent training samples. This preliminary action on the spectral data prepares the information in a form that maximizes learning efficiency from limited training pairs.
Data Source
AI summary
A messaging system for audio character type swapping. Methods of audio character type swapping include receiving input audio data having a first characteristic and transforming the input audio data to an input image where the input image represents the frequencies and intensities of the audio. The methods further include processing the input image using a convolutional neural network (CNN) to generate an output image and transforming the output image to output audio data, the output audio data having a second characteristic. The input audio and output audio may include vocals. The first characteristics may indicate a male voice and the second characteristics may indicate a female voice. The CNN is trained together with another CNN that changes input audio having the second characteristic to audio having the first characteristic. The CNNs are trained using discriminator CNNs that determine whether audio has a first characteristic or a second characteristic.