End-to-End Spoken Language Understanding System Using Unpaired Text Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
End-to-end spoken language understanding systems face challenges in training due to the scarcity of intent-labeled speech data, limiting their performance compared to traditional cascaded systems, and struggle to effectively leverage abundant text resources for improved intent classification.
Innovation Solution
The method involves training an end-to-end SLU system using unpaired text data with semantic labels by first training a text-to-intent model and then a speech-to-intent model, or generating synthetic speech from text using a TTS system to augment the training data, allowing the system to learn from annotated text without paired speech.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If end-to-end SLU systems are trained using traditional methods requiring paired speech and semantic annotation data, then the system can directly optimize intent recognition, but the training data becomes scarce and collection becomes time-consuming and expensive
Solution Approach 1:
The patent creates synthetic speech data by copying text data and generating corresponding audio representations. Text-to-speech synthesis is used to generate synthetic speech clips from text data, which are then used to train the acoustic model. This copying approach allows the system to leverage abundant text data to compensate for the scarcity of paired speech-annotation data.
Solution Approach 2:
The patent performs preliminary training of text embedding models and acoustic models separately before combining them. The text embedding model is pre-trained on abundant text data, and the acoustic model is pre-trained on synthetic speech data. This preliminary action allows each component to be optimized independently before integration, improving overall system performance despite data limitations.
2Quantity of substance
If abundant text resources are leveraged for training, then text-to-intent classification can be improved, but the challenge remains in effectively integrating text resources into end-to-end speech-to-intent systems
Solution Approach 1:
The patent segments the SLU system into distinct components: a text embedding model, an acoustic model, and an intent classification layer. Each component can be trained independently on appropriate data types. The text embedding model processes text data, the acoustic model processes speech data, and their features are combined later. This segmentation allows effective integration of text resources without overwhelming system complexity.
Solution Approach 2:
The patent introduces text embeddings as an intermediary representation that bridges text and speech modalities. The text embedding model converts text into numerical representations that can be processed by the acoustic model. This intermediary allows the system to leverage text resources while maintaining the end-to-end speech-to-intent architecture.
3Ease of operation
If paired speech and semantic annotation data are collected to train E2E SLU systems, then direct intent extraction from speech is enabled, but the data collection process becomes time-consuming and expensive
Solution Approach 1:
The patent creates synthetic speech data by copying text data and generating corresponding audio representations through text-to-speech synthesis. This allows the system to use abundant text data as a proxy for scarce paired speech data, eliminating the need for time-consuming manual speech data collection and annotation while maintaining the ability to extract intent directly from speech inputs.
Data Source
AI summary
An illustrative embodiment includes a method for training an end-to-end (E2E) spoken language understanding (SLU) system. The method includes receiving a training corpus comprising a set of text classified using one or more sets of semantic labels but unpaired with speech and using the set of unpaired text to train the E2E SLU system to classify speech using at least one of the one or more sets of semantic labels. The method may include training a text-to-intent model using the set of unpaired text; and training a speech-to-intent model using the text-to-intent model. Alternatively or additionally, the method may include using a text-to-speech (TTS) system to generate synthetic speech from the unpaired text; and training the E2E SLU system using the synthetic speech.