Hybrid Arbitration for Speech Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Hybrid automatic speech recognition and natural language understanding systems face challenges in selecting the best results between on-device and cloud-based processing due to differing computational resources and access to personal data, often relying solely on ASR confidence features and application-specific heuristics.
Innovation Solution
A method that uses ASR confidence features in conjunction with raw ASR strings and NLU features in a hybrid arbitration scheme to select the best speech recognition result from on-device and cloud-based processing outputs, employing a neural network classifier to compute confidence values and compare them against thresholds for decision-making.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If hybrid arbitration schemes use only ASR confidence features and application-specific heuristics to select between on-device and cloud-based results, then the arbitration process is simple and fast, but the selection accuracy is insufficient and cannot capture subtle context-specific differences
Solution Approach 1:
The patent merges multiple feature types (ASR confidence features, raw ASR strings, and NLU features) into a unified arbitration input set. This combination allows the arbitration scheme to capture both confidence metrics and contextual semantic information, significantly improving selection accuracy while maintaining a manageable processing framework through the use of neural network classifiers that can efficiently handle the integrated feature set.
Solution Approach 2:
The patent introduces neural network classifiers as intermediary components that process the combined feature set and generate arbitration decisions. These classifiers act as mediators between the raw input features and the final selection decision, transforming multiple heterogeneous features into a coherent arbitration outcome. This intermediary layer enables sophisticated decision-making without requiring complex rule-based logic throughout the entire system.
2Reliability
If the system waits for cloud-based processing results to ensure accuracy, then the ASR error rate decreases, but the latency increases and user experience deteriorates
Solution Approach 1:
The patent performs preliminary arbitration using available on-device results and features before cloud processing completes. The neural network classifier evaluates the confidence and quality of on-device results in advance, making preliminary selection decisions that can be immediately executed. This preliminary action reduces waiting time while maintaining accuracy by only deferring to cloud results when the preliminary assessment indicates lower confidence.
Solution Approach 2:
The arbitration scheme dynamically adjusts its behavior based on real-time confidence assessments. When on-device processing shows high confidence, the system immediately selects those results without waiting for cloud processing. When confidence is lower, the system dynamically switches to waiting for cloud results. This dynamic adaptation allows the system to optimize between speed and accuracy on a per-query basis rather than following a fixed protocol.
3Reliability
If on-device processing uses fewer computing resources, then the device operates independently with better privacy, but the ASR and NLU result quality is reduced compared to cloud-based processing
Solution Approach 1:
The patent applies local quality by using different processing approaches for different parts of the speech recognition pipeline. On-device processing handles initial ASR transcription with basic features, while cloud-based processing supplements with enhanced NLU analysis when needed. The arbitration system selectively applies cloud resources only to specific cases where quality improvement is necessary, rather than uniformly applying cloud processing to all inputs. This localized application of computational resources optimizes the balance between quality and energy consumption.
Data Source
AI summary
A method for selecting a speech recognition result on a computing device includes receiving a first speech recognition result determined by the computing device, receiving first features, at least some of the features being determined using the first speech recognition result, determining whether to select the first speech recognition result or to wait for a second speech recognition result determined by a cloud computing service based at least in part on the first speech recognition result and the first features.