Low-latency Speech Enhancement via Coarse-to-Fine FFT Transform
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern speech enhancement techniques for personal voice amplification products (PVAPs) face challenges in reducing power consumption and latency while maintaining accurate inferencing results, as they often require high frequency resolution for neural networks, leading to high processing power consumption and latency.
Innovation Solution
The approach involves transforming multiple frames of an audio signal in the coarse frequency domain to simulate a signal in the fine frequency domain, allowing for effective speech enhancement while meeting low latency and low power consumption constraints by sampling at a certain rate associated with the coarse frequency domain and transforming it to a signal in the fine frequency domain.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If neural networks use high frequency resolution for speech enhancement, then inferencing accuracy is improved, but power consumption and latency increase
Solution Approach 1:
The patent divides the frequency domain processing into multiple frames (B frames) that are processed separately through FFT transformations. By segmenting the processing into manageable frame units rather than processing the entire signal at once, the system achieves accurate inferencing while controlling computational load and power consumption per processing cycle.
Solution Approach 2:
The system dynamically adjusts the processing approach by transforming B*N time-domain samples into frequency-domain samples using FFT, then determining speech probability based on these transformed samples. This dynamic transformation allows the system to adapt processing intensity to actual speech conditions, optimizing the balance between accuracy and power consumption.
2Measurement precision
If neural networks use high frequency resolution for speech enhancement, then inferencing accuracy is improved, but latency increases
Solution Approach 1:
The patent segments the audio signal into B frames with N samples each, processing them through FFT transformations in manageable units. This segmentation enables the system to achieve high frequency resolution for accurate speech detection while limiting the processing time per frame, thereby controlling overall latency.
Solution Approach 2:
The system performs preliminary FFT transformations on the B*N time-domain samples before neural network inferencing. By pre-processing the signal into frequency-domain samples, the system prepares optimized input data that accelerates the subsequent neural network processing, reducing overall inferencing latency while maintaining accuracy.
3Object-affected harmful factors
If speech enhancement processes all environmental sounds, then noise suppression is improved, but speech distortion increases
Solution Approach 1:
The patent applies different processing characteristics to different frequency components and time frames. By determining speech probability locally for each frame and frequency bin rather than applying uniform processing, the system can suppress noise in specific regions while preserving speech characteristics in other regions, achieving noise suppression without excessive speech distortion.
Solution Approach 2:
The system uses the determined speech probability as feedback to control the enhancement process. The speech probability information guides the spectral suppression gain or filter application, allowing the system to adaptively preserve speech components while suppressing noise, thereby reducing speech distortion while maintaining noise suppression effectiveness.
Data Source
AI summary
This disclosure provides methods, devices, and systems for audio signal processing. The present implementations more specifically relate to low-latency speech enhancement. In some aspects, a speech enhancement system may receive a number (B) of frames of a signal, where each of the B frames include a number (N) of time-domain samples. The speech enhancement system may transform the B*N time-domain samples into B*N first frequency-domain samples based on an N-point fast Fourier transform (FFT), and may further transform the B*N first frequency-domain samples into B*N second frequency-domain samples based on a B-point FFT. The speech enhancement system may determine a probability of speech in the signal based at least in part on the B*N second frequency-domain samples. In some implementations, the speech enhancement system may decimate the B*N second frequency-domain samples by a factor (D), and the probability of speech is determined based on the B*N/D decimated second frequency-domain samples.