Hierarchical Filter-Bank Audio Generation With Built-In Phase Handling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing generative neural network models for audio processing face challenges in integrating with frequency-domain tools, require phase reconstruction, and have limited parallelization potential, especially when operating in the time-domain or on spectrograms.
Innovation Solution
A generative neural network model operating in the filter-bank domain that learns to generate filter-bank representations of audio signals, enabling easier integration with frequency-domain tools, suppressing quiet bands, and handling both magnitude and phase inherently, with increased parallel processing capabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If generative models operate in the time-domain (e.g., WaveNet, sampleRNN), then they can generate audio waveforms directly, but they cannot easily integrate with frequency-domain tools and have limited parallelization potential
Solution Approach 1:
The patent transforms the generative model from operating in the time-domain to operating in the filter-bank domain (frequency-domain). This dimensional change allows the model to process filter-bank representations directly, enabling seamless integration with frequency-domain tools while maintaining parallelization capabilities through the hierarchical structure that processes multiple filter-bank channels simultaneously.
Solution Approach 2:
The patent introduces a filter-bank representation as an intermediary between the generative model and the audio signal. Instead of directly processing time-domain waveforms, the model processes filter-bank representations, which serve as a bridge that enables both frequency-domain tool integration and efficient parallel processing through the hierarchical architecture.
2Ease of operation
If generative models operate on spectrograms (e.g., MelNet), then they can process frequency representations, but they require phase reconstruction algorithms as post-processing
Solution Approach 1:
The patent extracts and processes both magnitude and phase information simultaneously in the filter-bank domain, rather than only processing magnitude spectra and requiring separate phase reconstruction. The hierarchical generative model directly generates filter-bank representations that inherently contain both magnitude and phase information, eliminating the need for post-processing phase reconstruction algorithms.
3Reliability
If recursive networks are used for audio generation, then they can model temporal dependencies, but they have limited potential for parallelization
Solution Approach 1:
The patent segments the audio processing into a hierarchical structure with multiple tiers, where each tier processes different temporal scales. This segmentation allows temporal dependencies to be modeled at multiple levels while enabling parallel processing within each tier, as the hierarchical structure processes different time scales independently and simultaneously.
Solution Approach 2:
The patent transitions from a purely sequential recursive approach to a hierarchical structure that adds a temporal scale dimension. By processing multiple time scales in parallel across different tiers and then combining results, the model achieves both temporal dependency modeling and parallelization potential through this dimensional extension.
Data Source
Figure 1~2
Figure 3a~3b
Figure 4a
AI summary
A neural network system is provided, implementing a generative model for autoregressively generating a distribution for a plurality of current filter-bank samples of an audio signal, wherein the current samples correspond to a current time slot, and each current sample corresponds to a channel of the filter-bank. The system includes a hierarchy of a plurality of neural network processing tiers ordered from a top to a bottom tier, each tier trained to generate conditioning information based on previous filter-bank samples and, for at least each tier but the top tier, also on the conditioning information from a tier higher up in the hierarchy, and an output stage trained to generate the probability distribution based on previous samples for one or more previous time slots and the conditioning information from the lowest processing tier.