Multi-tier classifier for voice activity detection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing voice activity detection (VAD) systems that incorporate multiple modalities, such as audio-visual VAD, fail to effectively fuse information from different features and classifier outputs, often requiring prior knowledge of acoustic SNR or predetermined models, and do not consider interactions between features, leading to suboptimal performance, especially in low SNR conditions.

Innovation Solution

An augmented multi-tier classification system that combines feature fusion and decision fusion by allowing independent classifiers to provide feedback, using raw fused features as inputs to a second classifier, which learns to apply rules dynamically based on input features, enabling instantaneous or near-instantaneous voice activity detection without requiring temporal context.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If feature concatenation or majority voting is used to fuse multi-modal features, then the system can process multiple modalities, but it requires prior knowledge of acoustic SNR or predetermined models of feature uncertainty

Engineering Contradiction:

Improvemulti-modal processing capabilityVSAvoidrequirement for prior knowledge and predetermined models

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system performs self-calibration by automatically learning feature uncertainty and acoustic SNR from training data without requiring prior knowledge. The uncertainty model is derived empirically from the training set, allowing the system to serve itself rather than relying on externally provided parameters

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system dynamically adjusts the weighting of different modalities based on learned feature uncertainty and acoustic SNR estimates. Instead of using fixed weights or predetermined models, the parameters are adaptively changed during operation based on the actual input conditions

Inventive Principle:

Principle #35Parameter changes

2Adaptability or versatility

If existing AV-VAD systems use feature fusion or decision fusion, then they can combine multiple modalities, but they fail to incorporate or consider features with classifier output

Engineering Contradiction:

Improvemulti-modality integrationVSAvoidclassification accuracy

Core Design Contradiction:

Adaptability or versatilityVSMeasurement precision

Solution Approach 1:

The system merges feature fusion and decision fusion approaches by incorporating both the original multi-modal features and the classifier outputs as inputs to a final classifier. This creates a unified framework that leverages both low-level feature information and high-level classification decisions

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The system incorporates feedback from individual classifiers by using their outputs as additional inputs to the final classification stage. This feedback mechanism allows the system to refine its decision by considering both raw features and classifier predictions

Inventive Principle:

Principle #23Feedback

3Device complexity

If existing approaches consider only a few multimodal features, then they can simplify processing, but they do not utilize the broad set of available information and fail to consider interactions between features

Engineering Contradiction:

Improveprocessing simplicityVSAvoidutilization of available information

Core Design Contradiction:

Device complexityVSLoss of information

Solution Approach 1:

The system is designed to handle a universal and broad set of features from multiple modalities simultaneously. The framework is not limited to specific feature types but can accommodate various audio, visual, and linguistic features, making it highly versatile in utilizing available information

Inventive Principle:

Principle #6Universality (Multi-functionality)

4Measurement precision

If audio-only VAD is used with high acoustic SNR, then excellent voice activity detection can be achieved, but performance decreases rapidly as acoustic SNR decreases

Engineering Contradiction:

Improvevoice activity detection accuracyVSAvoidsensitivity to acoustic SNR

Core Design Contradiction:

Measurement precisionVSObject-affected harmful factors

Solution Approach 1:

The system uses a composite approach by combining multiple modalities (audio, visual, and linguistic features) rather than relying on a single modality. This composite framework provides robustness against poor acoustic conditions by leveraging complementary information from other modalities

Inventive Principle:

Principle #40Composite materials

Data Source

PatentUS9892745B2Augmented multi-tier classifier for multi-modal voice activity detection

Publication Date: 2018.02.13 AT&T INTELLECTUAL PROPERTY I L P

AI summary

Disclosed herein are systems, methods, and computer-readable storage media for detecting voice activity in a media signal in an augmented, multi-tier classifier architecture. A system configured to practice the method can receive, from a first classifier, a first voice activity indicator detected in a first modality for a human subject. Then, the system can receive, from a second classifier, a second voice activity indicator detected in a second modality for the human subject, wherein the first voice activity indicator and the second voice activity indicators are based on the human subject at a same time, and wherein the first modality and the second modality are different. The system can concatenate, via a third classifier, the first voice activity indicator and the second voice activity indicator with original features of the human subject, to yield a classifier output, and determine voice activity based on the classifier output.