Speech Recognition Model Language Modeling via Text Mapping Layer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transducer-based speech recognition models have insufficient language modeling ability, with the language predictor primarily functioning to eliminate duplicate labels rather than performing real reasoning, limiting their ability to model language dependencies.
Innovation Solution
The method involves training a speech recognition model with an acoustic encoder, a language predictor, a text mapping layer, and a joint network, using iterative optimization with cross-entropy and Transducer loss functions to improve language modeling capabilities, incorporating a text mapping layer to enhance the prediction process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a Transducer-based speech recognition model is used, then streaming speech recognition capability is improved, but language modeling ability deteriorates
Solution Approach 1:
The speech recognition model is segmented into distinct functional modules: acoustic encoder for processing audio inputs, language predictor for generating text predictions, text mapping layer for probability distribution, and joint network for loss calculation. This segmentation allows each component to specialize in its function while maintaining overall system performance for both streaming recognition and language modeling
Solution Approach 2:
The language predictor acts as an intermediary component between the acoustic encoder and the output layer. It receives acoustic features and generates text prediction vectors that are then processed by the text mapping layer, enabling the model to maintain streaming capability while improving language modeling through the intermediate reasoning step
2Measurement precision
If the language predictor is used to eliminate duplicate labels, then label accuracy is improved, but language dependency modeling deteriorates
Solution Approach 1:
The language predictor operates in a new dimensional space by generating text prediction vectors that capture linguistic patterns beyond simple label elimination. The text mapping layer then transforms these vectors into probability distributions over the vocabulary, adding a dimensional transformation that enables sophisticated language dependency modeling while maintaining label accuracy
Data Source
AI summary
Disclosed are a method for training speech recognition model, a method and a system for speech recognition. The disclosure relates to field of speech recognition and includes: inputting an audio training sample into the acoustic encoder to represent acoustic features of the audio training sample in an encoded way and determine an acoustic encoded state vector; inputting a preset vocabulary into the language predictor to determine text prediction vector; inputting the text prediction vector into the text mapping layer to obtain a text output probability distribution; calculating a first loss function according to a target text sequence corresponding to the audio training sample and the text output probability distribution; inputting the text prediction vector and the acoustic encoded state vector into the joint network to calculate a second loss function, and performing iterative optimization according to the first loss function and the second loss function.