Multimodal SLU Using Audio and ASR Text for Semantic Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing Spoken Language Understanding (SLU) systems face challenges in accurately determining semantic representations of spoken utterances, particularly in specific domains, due to the limitations of using only textual data from Automatic Speech Recognition (ASR) modules, which fail to leverage the rich semantic information present in audio data, such as non-verbal cues like prosody and intonation.
Innovation Solution
Fine-tuning a pre-trained Large Language Model (LLM) using both textual and audio data, incorporating a pre-trained audio encoder for self-supervised training, allows the model to leverage audio data to capture additional semantic information, improving the accuracy of semantic representation in specified domains.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If only textual data from ASR is used for fine-tuning the LLM, then the training process is simpler and faster, but the semantic representation accuracy is reduced due to loss of audio information
Solution Approach 1:
The patent merges audio data and textual data into a unified training framework for the LLM. The audio encoder extracts features from raw audio, which are then combined with ASR transcripts to create enriched training samples. This allows the model to learn from both audio and text modalities simultaneously, improving semantic representation accuracy while maintaining a manageable training process through modular architecture design.
2Measurement precision
If a domain-specific ASR module is trained from scratch, then domain accuracy is improved, but computational resources and time required increase significantly
Solution Approach 1:
The patent applies preliminary action by pre-training a general-purpose ASR module and audio encoder on diverse datasets before domain-specific fine-tuning. The audio encoder is pre-trained on generic audio features, and the ASR model is pre-trained on general speech patterns. This preliminary training establishes a robust foundation that reduces the computational resources and time needed for subsequent domain-specific adaptation, as the model only needs to learn domain-specific patterns rather than training from scratch.
Solution Approach 2:
The patent utilizes parameter changes by adapting pre-trained models through fine-tuning with domain-specific data. The LLM and ASR module are fine-tuned with domain-specific training data, which adjusts the model parameters to capture domain-specific patterns and semantics. This approach maintains the general capabilities of the pre-trained model while adapting it to specific domains, achieving domain accuracy without requiring complete retraining.
3Reliability
If audio data is incorporated into LLM fine-tuning, then semantic understanding of spoken utterances is improved, but training time and computational cost increase
Solution Approach 1:
The patent applies segmentation by dividing the audio processing into separate modules: an audio encoder that processes raw audio into features, and an LLM that processes the combined audio features and ASR transcripts. This modular segmentation allows independent optimization of each component and enables efficient training by processing audio and text in parallel, reducing overall training time while maintaining high reliability in spoken utterance understanding.
Data Source
Figure 1A
Figure 1B
Figure 2
AI summary
In various implementations, a method implemented by one or more processors of a computing device can comprise receiving audio data that captures a spoken utterance of a user; processing the audio data using an automatic speech recognition (ASR) model to generate textual data corresponding to the spoken utterance; generating a semantic representation corresponding to the spoken utterance of the user based on applying both the audio data and the textual data as input across a large language model (LLM); and causing the semantic representation corresponding to the spoken utterance of the user to be utilized in fulfilling the spoken utterance.