Convolutional Neural Network Waveform Synthesis from Spectrograms
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current algorithms for spectrogram inversion, such as the Griffin-Lim algorithm, are inefficient and inflexible, requiring many sequential iterations and lacking trainable parameters to improve perceptual audio quality, while autoregressive models for waveform synthesis are computationally intensive and challenging for deployment.
Innovation Solution
A convolutional neural network architecture that synthesizes waveforms from spectrograms without autoregressive computation, using a combination of audio reconstruction losses and a generative adversarial network framework, with multi-head convolutional neural networks to enhance inference efficiency and quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If the Griffin-Lim algorithm is used for spectrogram inversion, then the algorithm is simple to implement, but the synthesis speed is slow due to sequential iterations
Solution Approach 1:
The patent replaces the iterative mechanical computation process of the Griffin-Lim algorithm with a deep neural network model that performs waveform synthesis in a single forward pass. The neural network learns the inverse mapping from spectrogram to waveform during training, eliminating the need for sequential iterative refinement and achieving real-time synthesis speeds while maintaining or improving audio quality.
2Manufacturing precision
If autoregressive models are used for waveform synthesis, then the audio quality is high, but the computational complexity is intensive and deployment is challenging
Solution Approach 1:
The patent segments the waveform synthesis problem into two distinct stages: (1) spectrogram generation using an autoregressive model, and (2) waveform synthesis using a separate neural network vocoder. This segmentation allows the complex autoregressive modeling to operate only in the spectrogram domain rather than sample-by-sample in the waveform domain, significantly reducing computational complexity while preserving audio quality through the specialized vocoder network.
3Adaptability or versatility
If generic spectrogram inversion techniques are used, then the algorithms have fixed objective functions, but they lack flexibility to adapt to particular domains like human speech
Solution Approach 1:
The patent introduces dynamic, learnable parameters within the neural network vocoder that adapt to different domains and conditions. The model incorporates trainable parameters that allow it to learn domain-specific characteristics during training on particular datasets (e.g., human speech), enabling flexible adaptation to different applications while maintaining a relatively simple fixed architecture. This dynamic adaptation occurs through parameter learning rather than structural changes.
Data Source
AI summary
For the problem of waveform synthesis from spectrograms, presented herein are embodiments of an efficient neural network architecture, based on transposed convolutions to achieve a high compute intensity and fast inference. In one or more embodiments, for training of the convolutional vocoder architecture, losses are used that are related to perceptual audio quality, as well as a GAN framework to guide with a critic that discerns unrealistic waveforms. While yielding a high-quality audio, embodiments of the model can achieve more than 500 times faster than real-time audio synthesis. Multi-head convolutional neural network (MCNN) embodiments for waveform synthesis from spectrograms are also disclosed. MCNN embodiments enable significantly better utilization of modern multi-core processors than commonly-used iterative algorithms like Griffin-Lim and yield very fast (more than 300× real-time) waveform synthesis. Embodiments herein yield high-quality speech synthesis, without any iterative algorithms or autoregression in computations.