Multimodal Embedding Correction for ASR Accuracy Gaps
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing automatic speech recognition (ASR) systems suffer from low accuracy due to overfitting on training data and lack of separate acoustic and language models, leading to difficulties in generalizing unseen data and generating implausible contextually relevant transcriptions when the best speech recognition hypothesis is phonetically dissimilar to the actual utterance.
Innovation Solution
Implement a shared audio-text encoder to generate encoded audio and textual representations, and a correction model with higher order feature representations to determine a loss, training the encoders based on this loss, and leverage context data to bias the ASR model with contextually relevant transcriptions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a shared audio-text encoder is used to generate encoded representations, then the system can leverage multimodal embeddings for correction, but the device complexity increases due to additional encoders and higher order feature processing
Solution Approach 1:
The patent merges audio and text processing into a single shared encoder architecture that generates both encoded audio representations and encoded textual representations. This unified approach allows the system to leverage multimodal embeddings for transcription correction while managing complexity through shared computational components rather than completely separate processing streams.
Solution Approach 2:
The patent introduces higher order feature representations that transform the original encoded representations into an enhanced feature space. This dimensional transformation allows the correction model to capture more complex patterns and relationships in the data, improving transcription accuracy by operating in a richer feature dimensionality.
2Measurement precision
If additional transcriptions are used to correct misrecognized terms, then ASR accuracy improves, but the processing time and latency increase
Solution Approach 1:
The system performs preliminary encoding of both audio and text inputs through the shared encoder before the correction stage. By pre-computing the encoded representations and higher order features, the system prepares the data in advance for the correction process, reducing the computational burden during the actual transcription correction and thereby minimizing additional latency.
Solution Approach 2:
The patent uses encoded textual representations as a copy or approximation of the expected transcription, which can be quickly compared against the audio input. This copying approach allows for rapid correction by leveraging pre-processed text representations rather than performing full speech-to-text processing during the correction phase.
Data Source
AI summary
A method includes receiving training data that includes a set of transcribed speech utterances where each respective transcribed speech utterance is paired with a corresponding transcription. For each respective transcribed speech utterance, the method includes generating an encoded audio representation and an encoded textual representation, generating a higher order audio feature representation for a corresponding encoded audio representation, generating a higher order textual feature representation for a corresponding encoded textual representation, and determining a loss for the respective transcribed speech utterance based on the higher order audio feature representation and the higher order textual feature representation. The method also includes training a speech encoder and a text encoder of a correction model based on the loss determined for each transcribed speech utterance of the set of transcribed speech utterances.