Vocal Character Swapping With Paired CNN Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for modifying vocals require significant computation and lack flexibility, and there is a scarcity of vocal pairs with changed characteristics for training neural networks, making it difficult to achieve satisfactory results.
Innovation Solution
A pair of neural networks is trained together, with feedback loops and discriminator networks to determine characteristics, using input-output and feature losses to improve convergence, and stochastic gradient descent for training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional methods are used to modify vocals, then the modification can be achieved, but the computational requirements become excessively high and the flexibility is limited
Solution Approach 1:
The patent segments the vocal modification task into multiple independent neural network components: a feature extraction network that processes audio input, a style transfer network that applies desired characteristics, and a reconstruction network that outputs the modified vocals. This segmentation allows each component to be optimized independently and processed efficiently, reducing overall computational burden while maintaining flexibility for various modification types.
Solution Approach 2:
The patent changes the parameter representation of vocals by transforming audio signals into a latent feature space where modifications can be applied more efficiently. By representing vocals in terms of separable parameters (content features and style features), the system can apply modifications by adjusting specific parameters in the latent space rather than processing the entire audio signal, significantly reducing computational requirements while enabling flexible control over different vocal characteristics.
2Adaptability or versatility
If neural networks are trained for vocal characteristic changes, then flexibility is improved, but the scarcity of vocal pairs with changed characteristics makes training difficult
Solution Approach 1:
The patent introduces an intermediary latent feature space that mediates between source vocals and target vocals with different characteristics. This latent space acts as a bridge where style transfer can occur without requiring direct pairs of vocals with specific characteristic changes. The intermediary representation allows the network to learn generalizable transformation patterns from diverse training data, enabling flexible modifications even when specific vocal pairs are scarce.
Solution Approach 2:
The patent creates a universal neural network architecture that can perform multiple types of vocal modifications through a single trained model. The style transfer network is designed to handle various characteristic changes (gender, age, emotion, accent) using the same underlying mechanism and training approach. This universality allows the system to achieve high flexibility with limited training data, as the network learns general transformation capabilities rather than requiring separate models for each modification type.
3Manufacturing precision
If more computational resources are allocated to vocal modification, then the quality of results improves, but the complexity of the system increases
Solution Approach 1:
The patent divides the complex vocal modification system into three specialized neural network modules: feature extraction, style transfer, and reconstruction. Each module has a specific function and can be optimized independently for quality while maintaining manageable complexity. This segmentation allows high-quality results through specialized processing in each stage without requiring the entire system to be overly complex, as each component focuses on a specific aspect of the transformation.
Solution Approach 2:
The patent extracts and separates the style characteristics from the content features in the latent representation space. By taking out the style information as a distinct component that can be independently manipulated and applied, the system achieves high modification quality without increasing overall system complexity. The extracted style features can be applied through simple parameter adjustments rather than requiring complex transformation logic, maintaining system simplicity while delivering high-quality results.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A messaging system for audio character type swapping. Methods of audio character type swapping include receiving input audio data having a first characteristic and transforming the input audio data to an input image where the input image represents the frequencies and intensities of the audio. The methods further include processing the input image using a convolutional neural network (CNN) to generate an output image and transforming the output image to output audio data, the output audio data having a second characteristic. The input audio and output audio may include vocals. The first characteristics may indicate a male voice and the second characteristics may indicate a female voice. The CNN is trained together with another CNN that changes input audio having the second characteristic to audio having the first characteristic. The CNNs are trained using discriminator CNNs that determine whether audio has a first characteristic or a second characteristic.