GAN Waveform Generation via Spectrogram and Lp Norm Loss
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for generating realistic audio waveforms in speech synthesis and text-to-speech applications face challenges due to the complexity of waveforms and the inefficacy of using discriminators, which struggle to evaluate waveforms effectively, leading to unrealistic outputs.
Innovation Solution
A neural network model within a generative adversarial network (GAN) is trained to generate waveforms from spectrograms, using a combination of discriminator-based and spectrogram-based loss measures to improve the realism of the output through a back-propagation process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If a discriminator is used to evaluate generated waveforms in a GAN, then the model can learn to generate more realistic waveforms, but the discriminator introduces potential errors because waveforms do not represent human hearing well and are high-fidelity data that are difficult to learn patterns from
Solution Approach 1:
The patent introduces an intermediary loss function (spectrogram loss or Lp norm loss) that acts as a mediator between the generator and the discriminator. This intermediary provides an additional evaluation pathway that is more reliable for waveforms, complementing the discriminator's perceptual evaluation. The intermediary loss calculates the difference between real and generated waveforms in the spectrogram domain or using Lp norm, providing a more accurate gradient signal for training without relying solely on the discriminator's potentially erroneous waveform evaluation.
2Productivity
If conventional spectrogram loss is used to train the model, then training can proceed, but small discrepancies are largely ignored which results in a faint and unrealistic buzzing in the final audio output
Solution Approach 1:
The patent changes the parameter used for calculating loss from conventional spectrogram loss to Lp norm loss (particularly L1 or L2 norm). This parameter change fundamentally alters how discrepancies are measured: instead of using spectrogram magnitude which can mask small errors, the Lp norm directly measures the amplitude difference between real and generated waveforms in the time domain, making small discrepancies visible and enforceable during training, thereby eliminating the buzzing artifact.
3Adaptability or versatility
If a trained model generates waveforms from spectrograms, then speech synthesis applications can benefit, but the training process is very sensitive to the form of input data and the choice of loss function
Solution Approach 1:
The patent implements a dual-feedback training mechanism where the model receives gradient signals from two sources: the discriminator (providing perceptual feedback about waveform realism) and the intermediary loss function (providing direct mathematical feedback on waveform accuracy). This combined feedback system makes the training process more robust to variations in input data form and loss function choices, as the two feedback pathways compensate for each other's weaknesses, reducing sensitivity while maintaining versatility across applications.
Data Source
AI summary
Systems, apparatuses, and methods directed to generating waveforms that can be used to produce realistic sounding speech when used in a speech synthesis or text-to-speech application. In some embodiments, this includes training a neural network model that is part of a generative adversarial network (GAN) to generate the waveform from an input spectrogram, while implementing a specific set of processing stages to determine loss. The loss is used with a back-propagation process to update the weights of the model during a training cycle. In one example use case, text is converted to a spectrogram and the spectrogram is converted to a waveform by a trained model. The output of the model may be used to drive a transducer that converts the waveform to audible sound.