GAN Waveform Generation via Spectrogram Loss
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional approaches to generating realistic waveforms for speech synthesis and text-to-speech applications face challenges due to the complexity of waveforms and their representation, leading to unrealistic audio outputs, as they often rely on generative models with discriminators that are not optimal for evaluating waveforms, which are complex and do not accurately represent human hearing.
Innovation Solution
The use of a generative adversarial network (GAN) with a specific set of processing stages to determine loss terms, including a spectrogram loss term, to train a neural network model that generates waveforms from input spectrograms, improving the realism of the output by converting inputs to spectrograms and using Fourier transforms for better representation and error calculation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If a generative model with a discriminator is used to generate waveforms, then the waveform generation process can be automated, but the realism of the generated waveforms deteriorates because the discriminator is not optimal for evaluating waveforms
Solution Approach 1:
The patent introduces a spectrogram as an intermediary representation between the input text and the output waveform. Instead of directly generating waveforms and evaluating them with a discriminator, the system converts waveforms to spectrograms for evaluation. This intermediary step allows the model to work with a representation that better captures human hearing characteristics, thereby improving waveform realism while maintaining automation.
Solution Approach 2:
The patent changes the evaluation parameter from direct waveform comparison to spectrogram comparison. By transforming the waveform into a spectrogram representation for loss calculation, the system uses a different parameter space that is more suitable for capturing auditory perceptions. This parameter change enables better evaluation of waveform realism without requiring manual intervention.
2Ease of operation
If waveforms are directly evaluated using a discriminator, then the evaluation process is simple, but the accuracy of error measurement deteriorates because waveforms do not accurately represent human hearing
Solution Approach 1:
The spectrogram serves as an intermediary that bridges the gap between simple evaluation and accurate measurement. It provides a representation that is both easy to process computationally and accurate in representing human hearing characteristics. The evaluation process remains automated and straightforward while achieving high measurement precision through the spectrogram transformation.
3Ease of manufacture
If conventional generative models are used for waveform generation, then the model training process is straightforward, but the generated waveforms lack realism and produce unnatural buzzing
Solution Approach 1:
The spectrogram acts as an intermediary representation that simplifies the training process while improving waveform naturalness. By converting waveforms to spectrograms for loss calculation, the model learns from a representation that captures essential auditory characteristics. This makes the training process remain relatively straightforward while eliminating unnatural buzzing and improving overall waveform quality.
Solution Approach 2:
The patent changes the training parameter from direct waveform loss to spectrogram loss. This parameter change allows the model to learn from a representation that better captures human hearing characteristics, resulting in more natural waveforms. The training process remains computationally feasible and straightforward while achieving superior waveform naturalness.
Data Source
AI summary
Systems, apparatuses, and methods directed to training a Generator that is part of a Generative Adversarial Network (GAN) to generate “realistic” examples of a distribution. In some embodiments, this includes training the Generator model to receive a tensor as an input and generate a distribution which is then converted to the frequency domain and used to determine a loss term. Similarly, an actual distribution is also converted to the frequency domain and used to determine a loss term. The loss terms, generated distribution, and actual distribution are provided to a Discriminator which generates loss terms used as part of a backpropagation or feedback mechanism to modify the operation of the Generator and/or Discriminator.