Shared-Encoder Speech-Text Training for Domain-Adaptive ASR
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional automatic speech recognition systems face challenges in integrating abundant linguistic information due to the lack of a dedicated language model and the high cost of collecting and transcribing speech data for domain adaptation, leading to inferior modeling capabilities and increased adaptation costs.
Innovation Solution
A hybrid transducer and attention-based encoder-decoder model is trained using a shared encoder to process both speech and text embeddings, allowing for joint optimization and domain adaptation with text-based training data, reducing the need for expensive speech data collection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If an end-to-end based modeling approach is used to integrate acoustic and linguistic representations in one model, then the system complexity is reduced, but the modeling capability for linguistic information deteriorates because the model must be trained with speech corpus only
Solution Approach 1:
The patent combines separate acoustic model and language model into a unified hybrid model that processes both speech and text inputs simultaneously. The acoustic model component handles speech embeddings while the language model component handles text embeddings, with both contributing to the same prediction head, thereby integrating representations without sacrificing linguistic modeling capability
Solution Approach 2:
The hybrid model is designed to accept multiple input types (speech embeddings and text embeddings) and process them through shared and dedicated pathways. This multi-functional architecture allows the same model to effectively handle both acoustic and linguistic information, making it adaptable to different input modalities while maintaining specialized processing capabilities
2Reliability
If separate dedicated models are used for acoustic and linguistic information, then the modeling capability for each is improved, but the system complexity increases
Solution Approach 1:
The patent merges separate acoustic and language models into a single hybrid architecture where both modeling capabilities coexist. The acoustic model processes speech embeddings through convolutional and transformer layers, while the language model processes text embeddings through transformer layers, with both feeding into a shared prediction mechanism, thereby reducing system complexity while preserving specialized capabilities
3Adaptability or versatility
If domain adaptation is performed using speech data, then the model adapts to new domains, but the cost of data collection and transcription increases significantly
Solution Approach 1:
The hybrid model's ability to accept both speech and text inputs enables domain adaptation through text data alone. The language model component can be trained on domain-specific text corpora without requiring corresponding speech recordings, allowing the system to adapt to new domains while avoiding the expensive speech data collection and transcription process
Solution Approach 2:
The model uses text embeddings as a proxy or copy of speech information for domain adaptation purposes. By training the language model component on text data from target domains, the system captures domain-specific linguistic patterns without needing actual speech samples, effectively copying the adaptive benefit from text to the overall speech recognition system
Data Source
AI summary
A method includes generating speech embeddings corresponding to a received speech input using a speech encoder. The method also includes generating multi-modal embeddings corresponding to the received speech input using a shared encoder. The method further includes generating conditioned multi-modal embeddings corresponding to the received speech input using a predictor. In addition, the method includes generating a text prediction corresponding to the received speech input based on the multi-modal embeddings and the conditioned multi-modal embeddings using a joint network.