RNN-T Speech Recognition With Integrated Target Speaker Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The application of target speaker extraction technology in RNN-T systems requires a large amount of calculation, leading to response delays in real-time speech recognition, compromising the advantage of real-time processing.
Innovation Solution
A speech recognition model incorporating a first voice conversion unit, a second voice conversion unit, a symbol conversion unit, an estimation unit, a loss calculation unit, and an update unit, utilizing multilayer neural networks to convert and estimate target speaker voices from mixed voices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If target speaker extraction technology is applied to RNN-T speech recognition system, then speaker extraction accuracy is improved, but calculation amount increases and real-time processing capability deteriorates
Solution Approach 1:
The patent merges the target speaker extraction function with the speech recognition model by integrating voice conversion units (first and second voice conversion units) directly into the RNN-T architecture. This allows the model to perform both speaker extraction and speech recognition in a unified framework, sharing computational resources and avoiding separate processing stages, thereby maintaining real-time capability while achieving accurate speaker extraction.
Solution Approach 2:
The speech recognition model is designed to perform multiple functions simultaneously: it processes mixed speech signals, extracts target speaker voices through the voice conversion units, and performs speech recognition. The model parameters are updated to handle both speaker extraction and recognition tasks, making the system multi-functional without requiring separate dedicated systems for each task.
2Measurement precision
If target speaker extraction technology is applied to RNN-T speech recognition system, then speaker extraction accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent combines the voice conversion functionality with the existing speech recognition model structure. The first voice conversion unit converts auxiliary speech signals, and the second voice conversion unit converts mixed speech signals, with both units integrated into the RNN-T architecture. This merging approach allows shared computation and parameter updates across multiple functions, reducing overall computational complexity compared to separate dedicated systems.
Data Source
AI summary
A speech recognition model learning apparatus includes a first voice conversion unit converting an auxiliary feature amount XA into an auxiliary intermediate feature amount HA, using a first multilayer neural network, a second voice conversion unit receiving, as inputs, HA and a mixed sound feature amount XM and converting the feature amounts into a target speaker intermediate feature amount HS using a second multilayer neural network, a symbol conversion unit converting a symbol feature amount c into an intermediate character feature amount C using a third multilayer neural network, an estimation unit receiving HS and C as inputs and calculating an output probability distribution Y using the neural network, a loss calculation unit receiving CT and Y as inputs and calculating a loss LRNN-T, and an update unit updating model parameters of the first and second voice conversion unit, the symbol conversion unit, and an estimation unit using LRNN-T.