Multi-task DNN for Speech Enhancement and Voice Activity Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Consumer electronic devices face challenges in isolating target speech from background noise and interference during voice commands, leading to reduced speech intelligibility and accuracy in speech recognition systems.
Innovation Solution
A deep neural network (DNN) model employing multi-task learning is used to jointly enhance target speech signals and estimate audio parameters like voice activity detection, directional information, and echo probability, integrating these functions within a shared architecture to improve noise suppression and computational efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If speech enhancement and VAD are performed separately in time or separate operations, then the processing pipeline is simpler, but the robustness and quality of the enhanced target speech signal is reduced
Solution Approach 1:
The patent combines speech enhancement and VAD into a single joint model that processes the mixed signal simultaneously. The shared encoder extracts common features for both tasks, while task-specific decoders generate enhanced speech and VAD outputs. This integration allows the model to leverage internal representations that indicate small voice activities during non-speech periods, forcing the model to learn more robust speech characteristics and achieve lower background noise in the enhanced speech signal.
2Measurement precision
If separate speech enhancement and VAD processing is used, then computational resources are saved, but speech recognition accuracy is reduced
Solution Approach 1:
The joint model performs multiple functions simultaneously - speech enhancement and VAD - within a single unified architecture. The shared encoder and task-specific decoders enable the system to extract speech features and detect voice activity from the same input signal processing path, improving speech recognition accuracy while managing computational resources efficiently through shared feature extraction.
3Loss of information
If joint multi-task learning is implemented, then speech intelligibility and robustness are enhanced, but model complexity increases
Solution Approach 1:
The joint model is segmented into distinct functional components: a shared encoder for feature extraction, and separate task-specific decoders for speech enhancement and VAD. This segmentation allows the model to handle multiple tasks simultaneously while maintaining clear functional boundaries, improving speech intelligibility through specialized processing paths without requiring a completely monolithic complex architecture.
Data Source
AI summary
Disclosed is a multi-task machine learning model such as a time-domain deep neural network (DNN) that jointly generate an enhanced target speech signal and target audio parameters from a mixed signal of target speech and interference signal. The DNN may encode the mixed signal, determine masks used to jointly estimate the target signal and the target audio parameters based on the encoded mixed signal, apply the mask to separate the target speech from the interference signal to jointly estimate the target signal and the target audio parameters, and decode the masked features to enhance the target speech signal and to estimate the target audio parameters. The target audio parameters may include a voice activity detection (VAD) flag of the target speech. The DNN may leverage multi-channel audio signal and multi-modal signals such as video signals of the target speaker to improve the robustness of the enhanced target speech signal.