Multi-Task Speech Detection for Deepfake and Replay Audio
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems struggle to reliably detect synthetic speech generated by AI, particularly DeepFakes, due to inadequate training datasets and architectures that fail to address both physical and logical attacks effectively.
Innovation Solution
A multi-task synthetic speech detection (SSD) framework that combines speaker classification and channel classification tasks, using a large dataset and neural networks to enhance detection robustness, including consideration of codec effects and noise, with parallel training across multiple processing units.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If models are trained on small datasets for specific tasks, then training complexity is reduced, but detection reliability and applicability to multiple attack types deteriorates
Solution Approach 1:
The patent implements a multi-task neural network architecture that simultaneously performs speaker verification, synthetic speech detection, and channel classification. This universal model processes audio inputs through shared feature extraction layers (CNNs, RNNs) that learn representations useful for all three tasks, eliminating the need for separate models trained on small datasets and achieving reliable detection across multiple attack types including replay attacks and deepfake synthesis
Solution Approach 2:
The patent combines multiple detection tasks (speaker verification, synthetic speech detection, channel classification) into a single unified neural network framework. By merging these tasks and training them jointly on a large dataset, the system achieves improved detection reliability for all tasks compared to training separate models on small datasets, while the shared architecture maintains manageable complexity
2Adaptability or versatility
If separate models are used for different attack types, then model specialization is improved, but system complexity and computational resources increase
Solution Approach 1:
The patent creates a universal neural network that handles multiple attack types (replay attacks, deepfake synthesis, TTS attacks) within a single architecture. The model performs speaker verification, synthetic speech detection, and channel classification simultaneously, providing comprehensive attack type coverage without requiring separate specialized models for each threat
Solution Approach 2:
The patent segments the detection system into distinct functional modules within the neural network: feature extraction layers (CNNs for spectral features, RNNs for temporal features), task-specific output layers for each detection objective, and shared intermediate layers that learn general audio representations. This segmentation allows the system to handle diverse attack types while maintaining organized complexity and efficient computation
3Measurement precision
If comprehensive feature extraction is performed, then detection accuracy is improved, but computational resource consumption increases
Solution Approach 1:
The patent merges multiple feature extraction operations into a single integrated neural network pipeline. The CNN and RNN layers simultaneously extract spectral, temporal, and speaker-specific features in one pass, and these features are reused across all three tasks (speaker verification, synthetic speech detection, channel classification), reducing redundant computation while maintaining high detection accuracy
Solution Approach 2:
The patent performs preliminary feature extraction at the beginning of the processing pipeline, creating comprehensive audio representations that are then fed to all task-specific classifiers. This preliminary action ensures that detection accuracy is maximized through thorough feature analysis, while the extracted features serve multiple purposes downstream, amortizing the computational cost across multiple detection objectives
Data Source
AI summary
Systems and methods for synthetic speech detection includes receiving an input sample comprising audio and extracting acoustic features corresponding to speech in the audio. The extracted acoustic features are processed using a plurality of neural networks to output abstracted features and generating a feature vector corresponding to the abstracted features using pooling. Training of an SSD task, a speaker classification task, and a channel classification task are performed at a same time, using the feature vector. Synthetic speech is detected using at least the trained SSD task.