Streaming Noise Suppression With TCN State Buffers for Low Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Latency in dynamic noise suppression processes for streaming audio, particularly in teleconferencing and online collaboration systems, exceeds the maximum allowable threshold, disrupting the user experience and consuming a significant portion of the total latency budget.
Innovation Solution
Implementing a convolutional neural network with internal state buffering to process smaller segments of streaming audio data, using temporal convolution networks (TCN) with depth-wise convolution layers and state buffers to retain previous states, thereby reducing latency while maintaining audio quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If dynamic noise suppression is applied to streaming audio, then audio quality is improved, but latency increases significantly
Solution Approach 1:
The audio signal is divided into overlapping frames of N samples each, with a hop size of M samples between consecutive frames. This segmentation allows the noise suppression algorithm to process smaller chunks of audio data independently, reducing the overall processing latency while maintaining audio quality through the overlapping frame structure.
2Reliability
If larger audio segments are processed, then noise suppression performance is improved, but latency increases
Solution Approach 1:
The system dynamically adjusts the frame size N and hop size M parameters to optimize the balance between noise suppression performance and latency. By making these parameters configurable and adaptable, the system can process smaller segments when low latency is critical while maintaining effective noise suppression through appropriate parameter selection.
3Loss of time
If processing threshold is reduced to meet latency requirements, then latency is reduced, but audio quality may be compromised
Solution Approach 1:
The system performs preliminary processing by dividing audio into overlapping frames before applying the full noise suppression algorithm. This preliminary segmentation allows for more efficient processing with reduced latency, while the overlapping frame structure ensures that no audio information is lost, thereby maintaining audio quality despite the reduced processing threshold.
Data Source
AI summary
Techniques are provided for dynamic noise suppression. A methodology implementing the techniques according to an embodiment includes generating a magnitude spectrum and a phase spectrum of an input audio signal comprising speech and dynamic noise. The method also includes employing a temporal convolution network (TCN) to generate a separation mask based on the magnitude spectrum. The TCN comprises depth-wise (DW) convolution layers, each DW convolution layer including a state buffer to store a number of previous states of the associated DW convolution layer. The number of stored previous states is based on a dilation factor of the associated DW convolution layer. The method further includes multiplying the separation mask with the magnitude spectrum to separate the speech from the dynamic noise to obtain a denoised magnitude spectrum. The method further includes reconstructing the input audio signal with reduced dynamic noise based on the denoised magnitude spectrum and the phase spectrum.