Joint Speech Language Model Adapter Mapping Audio to Text
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current speech recognition systems using large language models (LLMs) face challenges in mapping speech to text without information loss and struggle with domain-specific entities not well-represented in training data, leading to potential misrecognitions and limited processing capabilities.
Innovation Solution
A joint speech and language model (SLM) that employs blank filtering to reduce speech data sequence length to match text token embedding space, combined with a domain-specific retriever to handle unique entities, and a speech adapter to map audio embeddings to textual embeddings within a pre-trained LLM, enabling accurate speech recognition and understanding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a shared encoder is used to map speech and text to the same latent representations, then the system can process both modalities, but speech information loss occurs during the mapping process
Solution Approach 1:
The system separates speech processing and text processing into distinct encoders rather than using a shared encoder. The speech encoder processes audio inputs while the text encoder processes textual inputs, with separate processing paths that converge only at the embedding level, thereby preventing information loss in either modality
Solution Approach 2:
A speech adapter module acts as an intermediary between the speech encoder and the text encoder. This adapter transforms speech embeddings into a format compatible with the text encoder's embedding space, enabling cross-modal processing while preserving the original speech information through the transformation process
2Ease of manufacture
If a cascaded approach with ASR followed by LLM is used, then the system can leverage existing LLM capabilities, but ASR misrecognitions cannot be corrected and domain-specific entities are not handled well
Solution Approach 1:
The system merges the speech recognition and language understanding functions into a single joint model. The speech encoder and text encoder are trained together with a shared loss function, allowing the model to simultaneously optimize for speech recognition accuracy and language understanding, thereby correcting ASR errors and improving domain-specific entity handling while maintaining LLM capabilities
Solution Approach 2:
The system modifies the training parameters and architecture to accommodate both speech and text processing. By introducing speech-specific parameters and training objectives alongside the existing text processing parameters, the model adapts to handle domain-specific entities effectively while preserving the underlying LLM capabilities
3Loss of information
If the full speech sequence is processed without filtering, then all speech information is preserved, but processing requirements and computational load increase significantly
Solution Approach 1:
The blank filter extracts and removes silent or non-informative frames from the speech sequence before processing. By identifying and eliminating these redundant frames, the system reduces the computational load and processing requirements while preserving all meaningful speech information that is necessary for accurate recognition
Data Source
AI summary
Methods and systems for recognizing speech are disclosed herein. A method can include performing blank filtering on a received speech input to generate a plurality of filtered encodings and processing the plurality of filtered encodings to generate a plurality of audio embeddings. The method can also include mapping each audio embedding of the plurality of audio embeddings to a textual embedding using a speech adapter to generate a plurality of combined embeddings and receiving one or more specific textual embeddings from a domain-specific entity retriever based on the plurality of filtered encodings. The method can further include providing plurality of combined embeddings and the one or more specific textual embeddings to a machine-trained model and receiving a textual output representing speech from the speech input from the machine-trained model.