Teacher-Student PIT Framework for Multi-Talker Speech Recognition

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current methods for multi-talker speech recognition, such as deep learning models, face challenges with high word error rates due to label ambiguity and permutation issues, especially in scenarios with multiple speakers and single-channel mixed speech.

Innovation Solution

The implementation of a Teacher-Student permutation invariant training (PIT) framework, which transfers knowledge from single-talker ASR models to multi-talker ASR models using soft labels and a progressive training scheme, along with data augmentation and domain adaptation, to improve recognition accuracy.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If deep learning models are used for multi-talker speech recognition, then recognition capability is improved, but word error rate increases due to label ambiguity and permutation issues

Engineering Contradiction:

Improverecognition capabilityVSAvoidword error rate

Core Design Contradiction:

Adaptability or versatilityVSMeasurement precision

Solution Approach 1:

The patent segments the multi-talker speech recognition problem by introducing separate output streams for different talkers, allowing the model to process mixed speech through multiple specialized pathways rather than a single ambiguous stream

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces an intermediary assignment mechanism that maps output streams to talkers based on permutation invariant training, serving as a mediator between the deep learning model's outputs and the ground truth labels to resolve label ambiguity

Inventive Principle:

Principle #24Intermediary (Mediator)

2Measurement precision

If speaker adaptation is applied to reduce mismatch between training and test speakers, then WER improves for single-talker cases, but it cannot be directly applied to multi-talker scenarios

Engineering Contradiction:

ImproveWERVSAvoidapplicability to multi-talker scenarios

Core Design Contradiction:

Measurement precisionVSAdaptability or versatility

Solution Approach 1:

The patent creates a universal PIT training framework that can handle both single-talker and multi-talker scenarios, making the adaptation mechanism versatile across different speech recognition contexts by treating talker assignment as a permutation-invariant problem

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Ease of manufacture

If traditional training methods are used for multi-talker ASR, then model training is simpler, but recognition accuracy remains low due to label permutation problems

Engineering Contradiction:

Improvetraining simplicityVSAvoidrecognition accuracy

Core Design Contradiction:

Ease of manufactureVSMeasurement precision

Solution Approach 1:

The patent enables the model to self-resolve the label permutation problem through permutation invariant training, where the training process automatically learns to match output streams to talkers without requiring external intervention or complex preprocessing

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS10699697B2Knowledge transfer in permutation invariant training for single-channel multi-talker speech recognition

Publication Date: 2020.06.30 TENCENT TECHNOLOGY (SHENZHEN) CO LTD

AI summary

Provided are a speech recognition training processing method and an apparatus including the same. The speech recognition training processing method includes acquiring a multi-talker mixed speech signal from a plurality of speakers, performing permutation invariant training (PIT) model training on the multi-talker mixed speech signal based on knowledge from a single-talker speech recognition model and updating a multi-talker speech recognition model based on a result of the PIT model training.