Incremental Natural Language Understanding for Low Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current Spoken Language Understanding (SLU) systems face a bottleneck in latency due to the sequential processing of automatic speech recognition (ASR) and natural language understanding (NLU) modules, where NLU has to wait for ASR to finish processing, leading to delayed responses in real-time applications.
Innovation Solution
Implementing incremental natural language understanding by generating speech recognition outputs for portions of an utterance, allowing intent determination and resource retrieval before completing the utterance, and incorporating these resources into media content in real-time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If sequential processing of ASR and NLU modules is used, then processing accuracy is maintained, but latency increases
Solution Approach 1:
The patent segments the utterance into multiple portions and processes them incrementally. The ASR module generates recognition results for portions of the utterance as they become available, and the NLU module processes these incremental results to determine intent. This segmentation allows both modules to operate in parallel rather than sequentially, reducing latency while maintaining processing accuracy.
Solution Approach 2:
The patent implements preliminary action by determining intent based on incremental speech recognition results before the utterance is completely processed. The NLU module can start determining intent and retrieving resources based on partial recognition results, allowing the system to respond faster without waiting for complete utterance processing.
2Loss of time
If NLU waits for ASR to finish processing, then processing completeness is ensured, but response time increases
Solution Approach 1:
The patent introduces dynamics by making the NLU processing adaptive to the availability of ASR results. Instead of waiting for complete ASR processing, the NLU module dynamically adjusts its processing based on incremental results becoming available. This allows the system to respond as soon as sufficient information is detected while maintaining the ability to refine processing as more utterance data becomes available.
3Loss of time
If incremental speech recognition is implemented, then latency is reduced, but computational complexity increases
Solution Approach 1:
The patent applies partial action by processing only the necessary portions of the utterance as they become available. Instead of waiting for the complete utterance and then processing everything at once, the system processes portions incrementally as they are detected, reducing the total computational load and latency. The system processes exactly what is needed at each step rather than preparing for all possible future processing requirements.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A system for incremental natural language understanding includes a media module, a memory storing a software code, and a hardware processor communicatively coupled to the media module. The hardware processor is configured to execute the software code to receive an audio stream including a first utterance, and generate a first and second incremental speech recognition outputs based on first and second portions of the first utterance. In addition, the hardware processor is configured to execute the software code to determine, prior to generating the second incremental speech recognition output, a first intent of the first utterance based on the first incremental speech recognition output. The hardware processor is further configured to execute the software code to retrieve a first resource based on the determined first intent, and incorporate the first resource in the media content to be played by the media module.