Band-Split Audio Transformers for Fine-Grained Source Separation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing audio source separation methods, particularly in frequency-domain, struggle to capture the different frequency patterns of signals, leading to suboptimal performance in tasks like music source separation.
Innovation Solution
The implementation of a band-split Transformer with Rotary Position Encoding (BS-RoFormer) that splits audio representations into finer subbands, applies individual MLP layers, and uses interleaved Transformers along the frequency and time axes, incorporating Rotary Position Encoding for improved positional awareness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If frequency-domain methods apply the same kernels across all frequencies, then the method is simple and computationally efficient, but it fails to capture the different frequency patterns of signals
Solution Approach 1:
The patent divides the frequency spectrum into multiple bands (e.g., low, mid, high frequency bands) and applies separate convolutional kernels to each band. This segmentation allows the system to capture different frequency patterns with band-specific kernels while maintaining computational efficiency through the use of standard convolution operations.
Solution Approach 2:
The patent assigns different kernel characteristics to different frequency bands, allowing each band to be processed with optimized local properties. For example, low-frequency bands may use kernels optimized for capturing temporal patterns, while high-frequency bands use kernels optimized for capturing spectral details, thereby improving overall frequency pattern capture.
2Measurement precision
If audio is split into finer subbands with individual MLP layers, then frequency pattern recognition improves, but computational complexity increases
Solution Approach 1:
The audio spectrum is divided into multiple subbands, each processed by dedicated MLP layers. This segmentation enables precise frequency pattern recognition in each band while allowing parallel processing of multiple subbands, which distributes computational load and prevents excessive memory usage.
Solution Approach 2:
The patent transforms the audio representation into the frequency domain using Fourier transforms, adding a frequency dimension to the temporal signal. This dimensional transformation allows MLP layers to operate on frequency patterns independently, improving recognition capability while maintaining computational tractability through efficient transform algorithms.
3Reliability
If interleaved Transformers are applied along frequency and time axes, then global information capture improves, but computational burden increases
Solution Approach 1:
The Transformer architecture is applied separately along the frequency axis and time axis in an interleaved manner, rather than computing the full 2D attention matrix simultaneously. This segmentation of the attention computation into sequential 1D operations reduces computational burden from O(N²) to O(N) while still capturing global dependencies in both dimensions.
Solution Approach 2:
The interleaved application of frequency-axis and time-axis Transformers creates a periodic computation pattern that alternates between capturing frequency relationships and temporal relationships. This periodic structure allows the model to progressively build global understanding through multiple passes, improving reliability while managing computational resources efficiently.
Data Source
AI summary
The present disclosure describes techniques for implementing improved audio source separation. A complex spectrum X is split into a plurality of K bands along a frequency axis by applying band-split operations on the complex spectrum X. The complex spectrum is a time-frequency representation of audio signals. Each of the plurality of K bands is denoted as Xk, k=1, . . . , K. Each band Xk comprises one or more frequency bins. Each individual multilayer perceptron is applied to each band Xk to extract latent representations and obtain outputs Hk0. A time-domain transformer and a frequency-domain transformer are applied on a stacked representation H0. Time-domain and frequency-domain transformers are repeatedly applying in an interleaved manner for L times to obtain HL output from the transformer blocks. The HL is input into a multi-band mask estimation sub-model. A complex ideal ratio mask is generated based on outputs from the multi-band mask estimation sub-model.