Hybrid Voice Command Processing with Local Cloud Arbitration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current on-device automatic speech recognition (ASR) systems face limitations in processing voice commands due to hardware constraints, requiring a limited vocabulary and complexity, and existing confidence measures perform poorly for small vocabulary systems, leading to inaccurate decisions when handling out-of-domain inputs.
Innovation Solution
A hybrid approach is implemented, where a likelihood ratio is calculated between in-domain and out-domain confidence scores to determine whether voice command processing should be performed locally on the device or remotely in the cloud, using a combination of Hidden Markov Models (HMMs) and Deep Neural Networks (DNNs, along with Weighted Finite State Transducers (WFST) graphs to filter and evaluate audio features.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If speech recognition is performed entirely on the device, then responsiveness and availability are improved, but hardware constraints limit vocabulary size and command complexity
Solution Approach 1:
The speech recognition system is segmented into two parts: a local command spotter that handles wake-word and simple commands for immediate responsiveness, and a cloud-based ASR system that handles complex vocabulary and commands. This segmentation allows each component to be optimized independently for its specific function.
Solution Approach 2:
A hybrid arbitration mechanism acts as an intermediary between the local command spotter and cloud ASR system. It receives input from both systems, evaluates their confidence scores, and selects the most reliable output, enabling the system to leverage both local speed and cloud versatility.
2Adaptability or versatility
If speech recognition is performed in the cloud, then vocabulary size and command complexity are improved, but latency and loss of availability occur
Solution Approach 1:
The local command spotter performs preliminary recognition of wake-words and simple commands before cloud processing is needed. This preliminary action filters out routine commands that would otherwise incur cloud latency, reserving cloud resources for more complex recognition tasks.
Solution Approach 2:
The system dynamically routes commands based on their complexity and the confidence scores from both local and cloud systems. Simple, high-confidence local recognitions are executed immediately, while ambiguous or complex commands are forwarded to the cloud, creating a dynamic response time adapted to each input.
3Adaptability or versatility
If hybrid arbitration is used to select between device and cloud ASR results, then versatility is improved, but confidence measures perform poorly for small vocabulary systems
Solution Approach 1:
Different confidence evaluation strategies are applied locally for the command spotter versus the cloud ASR system. The command spotter uses confidence measures optimized for small vocabulary recognition, while the cloud system uses measures appropriate for large vocabulary recognition. This local quality approach ensures each system's confidence measures are accurate for their specific domain.
Solution Approach 2:
The arbitration mechanism adjusts decision thresholds and weighting parameters based on the type of input and system state. For example, different thresholds are used for wake-word detection versus full command recognition, and for different noise conditions. This parameter adaptation improves the accuracy of hybrid arbitration across diverse scenarios.
Data Source
AI summary
Digitized audio command is decoded to generate audio features. An in-domain confidence score is calculated for a model trained by a limited set of peripheral device commands. An out-domain confidence score is calculated for a model trained without the peripheral device commands. The best score determines whether to process the audio locally or at a remote server. In some embodiments, a likelihood ratio (LR) is calculated of the in-domain and out-domain confidence scores. Based on the likelihood ratio, a locally decoded audio command is performed, or the audio features are sent to a remote server for processing to determine the audio command.