Time-Domain Speech Extraction Using Machine Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Electronic devices with multiple microphones face challenges in enhancing user speech signals due to interference from bystanders and background noise, degrading performance in applications like assistive hearing, telephony, and augmented reality.
Innovation Solution
A machine learning model is trained to process time-domain audio signals from multiple microphones, optimizing a cost function to filter out non-user audio data and enhance user speech, using a combination of analysis, extraction, and synthesis networks to output an improved time-domain waveform.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple microphones are used to capture audio signals, then the ability to enhance user speech is improved, but interference from bystanders and background noise worsens
Solution Approach 1:
The audio signal is segmented into multiple frequency bands using filter banks, allowing different processing strategies for different frequency ranges. This enables selective enhancement of speech frequencies while suppressing noise frequencies, resolving the contradiction between capturing clear speech and rejecting noise interference.
Solution Approach 2:
A machine learning model acts as an intermediary between the raw microphone signals and the final enhanced output. The model processes the mixed signals from multiple microphones, learns to distinguish speech from noise patterns, and produces separated output signals, thereby resolving the interference problem while preserving speech quality.
2Measurement precision
If traditional signal processing methods are used, then device complexity is reduced, but speech enhancement performance deteriorates
Solution Approach 1:
Traditional mechanical signal processing methods (filtering, beamforming) are replaced with a data-driven machine learning approach. The neural network learns optimal signal separation strategies from training data, achieving superior speech extraction accuracy without relying on hand-crafted processing pipelines, thus resolving the contradiction between performance and complexity.
Solution Approach 2:
The system transforms the audio signals from time-domain to frequency-domain representations using Fourier transforms, and processes them in different signal representations (spectrograms, mel-frequency features). These parameter transformations enable the machine learning model to capture speech patterns more effectively, improving extraction accuracy while the transformations are computationally efficient.
3Object-affected harmful factors
If frequency-domain processing is used, then noise filtering is improved, but time-domain speech signal quality worsens
Solution Approach 1:
The system uses short-time Fourier transforms with overlapping windows, processing the signal in periodic segments. This allows frequency-domain noise filtering to be applied locally in time, and the results are combined through overlap-add to reconstruct the time-domain signal, thereby maintaining both noise reduction and time-domain speech quality.
Solution Approach 2:
The machine learning model serves as an intermediary that operates in the frequency domain to filter noise, then transforms the processed signals back to the time domain. The model learns to preserve speech temporal characteristics during frequency-domain processing, ensuring that the final time-domain output maintains high speech quality while benefiting from frequency-domain noise reduction.
Data Source
AI summary
A device for processing audio signals in a time-domain includes a processor configured to receive multiple audio signals corresponding to respective microphones of at least two or more microphones of the device, at least one of the multiple audio signals comprising speech of a user of the device. The processor is configured to provide the multiple audio signals to a machine learning model, the machine learning model having been trained based at least in part on an expected position of the user of the device and expected positions of the respective microphones on the device. The processor is configured to provide an audio signal that is enhanced with respect to the speech of the user relative to the multiple audio signals, wherein the audio signal is a waveform output from the machine learning model.