Partial-Hypothesis Streaming Action Fulfillment Before Speech Endpoint
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing digital assistant interfaces on mobile devices experience user perceived latency in fulfilling user queries due to the need to wait for the endpoint of an utterance before initiating sub-actions, despite advancements in on-device speech recognition that reduce latency for transcription.
Innovation Solution
Implementing a system that generates intermediate speech recognition results during an utterance to perform sub-actions related to the final action before the user finishes speaking, leveraging partial hypotheses to initiate actions in real-time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the system waits for the endpoint of an utterance before initiating sub-actions, then the accuracy of speech recognition is improved, but the user perceived latency increases
Solution Approach 1:
The system performs preliminary speech recognition processing during the utterance to generate intermediate results and partial hypotheses before the endpoint is reached. This allows sub-actions to be initiated in advance based on incomplete but progressively improving speech recognition data, thereby reducing user perceived latency while maintaining acceptable accuracy through continuous refinement.
Solution Approach 2:
The system performs speech recognition and action initiation based on partial speech data rather than waiting for complete utterances. By accepting intermediate speech recognition results with varying confidence levels, the system can start processing sub-actions earlier, trading off some precision for reduced latency, and then refine or correct actions as more complete speech data becomes available.
2Loss of time
If the system performs speech recognition on-device, then the latency for transcription is reduced, but the reliability of action fulfillment is worsened due to slow or unreliable local applications
Solution Approach 1:
The system introduces an intermediary layer of cloud-based services that receives intermediate speech recognition results and coordinates the execution of sub-actions across multiple applications. This intermediary cloud service acts as a reliable orchestrator that can manage complex action sequences involving multiple applications, compensating for the unreliability of individual local applications while preserving the low-latency benefits of on-device speech recognition.
Solution Approach 2:
The system segments the action fulfillment process into multiple independent sub-actions that can be executed in parallel or sequence. By breaking down the overall task into smaller, manageable sub-actions, the system can distribute execution across different local applications and cloud services, improving overall reliability through modular execution while maintaining the speed advantages of on-device processing.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
A method (300) for streaming action fulfillment receives audio data (14) corresponding to an utterance (20) where the utterance includes a query (22) to perform an action (24) that requires performance of a sequence of sub-actions (26) in order to fulfill the action. While receiving the audio data, but before receiving an end of speech condition (214), the method processes the audio data to generate intermediate automated speech recognition (ASR) results (212), performs partial query interpretation on the intermediate ASR results to determine whether the intermediate ASR results identify an application type needed to perform the action and, when the intermediate ASR results identify a particular application type, performs a first sub-action in the sequence of sub-actions by launching a first application (118) to execute on the user device where the first application is associated with the particular application type. The method fulfills performance of the action.