ASR Model Adaptation via Textogram Integration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
End-to-end automatic speech recognition (ASR) models face limitations in domain adaptation, requiring paired speech and text transcripts for training, which restricts their ability to adapt to new domains using only text data, and existing methods are costly or require additional processing steps.
Innovation Solution
The method trains ASR models using both spectrogram and textogram features, allowing adaptation to new domains with text-only data by integrating text inputs into the training process, enabling customization of both prediction and joint networks without needing transcribed speech.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If E2E ASR models are trained using only paired speech and text transcripts, then the model achieves end-to-end speech recognition capability, but the model cannot be adapted to new domains using text-only data
Solution Approach 1:
The patent creates a unified model architecture that can process both speech inputs (spectrograms) and text inputs (textograms) through the same prediction and joint networks. This multi-functionality allows the single model to serve both ASR and text-based adaptation purposes, eliminating the need for separate models for speech and text processing
Solution Approach 2:
The patent introduces textograms as an intermediary representation that bridges text data and the speech-based model architecture. By converting text into a format (textograms) that resembles spectrograms in structure and dimensionality, the model can process text inputs through the same computational pathways designed for speech, enabling domain adaptation without requiring actual speech recordings
2Adaptability or versatility
If hybrid ASR models use separate acoustic model and language model components, then each component can be trained on independent data sets, but the system complexity increases
Solution Approach 1:
The patent merges the acoustic model and language model into a single unified end-to-end model. The prediction network and joint network simultaneously handle both acoustic pattern recognition and language modeling functions, eliminating the need for separate components while maintaining the ability to learn from both speech and text data through a unified training objective
3Reliability
If ASR models require transcribed speech data for domain adaptation, then adaptation accuracy is maintained, but the cost and processing requirements increase significantly
Solution Approach 1:
The patent creates a simplified copy or approximation of speech data in the form of textograms. Instead of requiring actual speech recordings for domain adaptation, the system generates text-based representations that mimic the structural properties of spectrograms, allowing the model to learn domain-specific patterns without the computational and logistical burden of collecting and processing real speech data
Data Source
AI summary
A processor-implemented method trains an automatic speech recognition system using speech data and text data. A computer device receives speech data, and generates a spectrogram based on the speech data. The computing device receives text data associated with an entire corpus of text data, and generates a textogram based upon the text data. The computing device trains an automatic speech recognition system using the spectrogram and the textogram.