Multi-Talker Speech Recognition via Permutation Invariant Training

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current methods for multi-talker speech recognition, such as deep learning models and techniques like deep clustering and deep attractor networks, still result in high word error rates due to label ambiguity and speaker switching issues in mixed speech scenarios, falling short of single-talker recognition accuracy.

Innovation Solution

The implementation of permutation invariant training (PIT) with an attention mechanism, which encodes mixed speech data using gated convolution operations, generates speaker-specific context vectors, and updates acoustic models based on senone posteriors to improve speech recognition accuracy in multi-talker environments.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If deep learning models are used for multi-talker speech recognition, then speech recognition capability is provided, but word error rates remain high due to label ambiguity and speaker switching issues

Engineering Contradiction:

Improvespeech recognition accuracyVSAvoidword error rate

Core Design Contradiction:

ReliabilityVSMeasurement precision

Solution Approach 1:

The patent segments the mixed speech signal into individual speaker streams by introducing a speaker embedding layer that maps mixed speech features to separate speaker-specific embeddings. This segmentation allows the model to process each speaker's contribution independently, resolving the label ambiguity problem where traditional models treated all speakers as a single mixed source.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces an intermediate embedding space as a mediator between the input mixed speech and the output recognition results. The speaker embedding layer creates a intermediate representation that separates different speakers' contributions before final recognition, allowing the model to handle speaker switching without direct mapping from mixed speech to labels.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Device complexity

If traditional speech recognition models are used, then processing is simpler, but they cannot handle label permutation problems in multi-talker scenarios

Engineering Contradiction:

Improvemodel structure simplicityVSAvoidmulti-talker speech handling capability

Core Design Contradiction:

Device complexityVSAdaptability or versatility

Solution Approach 1:

The patent adds a new dimensional layer to the model by introducing speaker embeddings that operate in an additional embedding space. This extra dimension allows the model to represent and differentiate multiple speakers simultaneously, providing adaptability to multi-talker scenarios while maintaining a relatively straightforward architectural extension from traditional models.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

Data Source

PatentUS10699700B2Monaural multi-talker speech recognition with attention mechanism and gated convolutional networks

Publication Date: 2020.06.30 TENCENT TECHNOLOGY (SHENZHEN) CO LTD

AI summary

Provided are a speech recognition training processing method and an apparatus including the same. The speech recognition training processing method includes acquiring multi-talker mixed speech sequence data corresponding to a plurality of speakers, encoding the multi-speaker mixed speech sequence data into an embedded sequence data, generating speaker specific context vectors at each frame based on the embedded sequence, generating senone posteriors for each of the speaker based on the speaker specific context vectors and updating an acoustic model by performing permutation invariant training (PIT) model training based on the senone posteriors.