Hybrid Speech Processing On-Device Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current speech processing systems face challenges in efficiently processing user commands on devices due to the computational intensity of automatic speech recognition (ASR), natural language understanding (NLU), and command execution, leading to high resource requirements and latency, especially when not connected to a remote system.
Innovation Solution
A hybrid speech processing system that performs on-device learning by comparing local NLU data to remote NLU data to improve a speech processing model, allowing devices to process voice commands locally while also sending data to a remote system for aggregation and model improvement, thereby reducing latency and improving performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If speech processing is performed entirely on-device, then processing speed and responsiveness improve, but computational resource requirements and model accuracy deteriorate
Solution Approach 1:
The system segments speech processing into two parts: on-device processing for speed-critical functions (wake word detection, initial command recognition) and cloud-based processing for accuracy-critical functions (complex NLU, model training). This segmentation allows each component to operate at its optimal performance level, resolving the contradiction between speed and accuracy.
Solution Approach 2:
The patent introduces an intermediary mechanism where the device sends speech data to a remote system for processing and receives updated model parameters in return. This intermediary cloud-based processing layer enables the on-device model to benefit from large-scale computational resources and data aggregation without requiring all processing to occur locally, thus maintaining accuracy while enabling fast local inference.
2Reliability
If speech processing models are updated frequently, then model accuracy improves, but device complexity and energy consumption increase
Solution Approach 1:
The system performs model updates in advance on the cloud platform, where computationally intensive training and optimization are completed before deployment. The device receives pre-trained model updates rather than performing training itself, which eliminates the energy cost of frequent local retraining while still benefiting from improved model accuracy.
Solution Approach 2:
Instead of copying large amounts of training data to the device for local processing, the system copies only the essential model parameters and updates. This selective copying approach allows the device to maintain an accurate model with minimal energy expenditure, as the heavy computational work is performed on the cloud and only compact update representations are transferred to the device.
3Reliability
If all speech data is processed remotely, then model accuracy improves, but latency and network dependency increase
Solution Approach 1:
The system segments processing tasks by urgency and complexity: time-critical tasks (wake word detection, basic command recognition) are handled locally on the device, while less time-sensitive tasks (complex NLU, model refinement) are processed remotely. This segmentation eliminates latency for critical functions while still leveraging remote processing for accuracy-critical functions.
Solution Approach 2:
The device maintains a local copy of the speech processing model and performs inference using this pre-loaded model, eliminating the need to fetch the model from the cloud for each processing task. This preliminary loading of the model enables fast local processing while periodic updates from the remote system ensure the model remains accurate.
Data Source
AI summary
A speech interface device is configured to receive response data from a remote speech processing system for responding to user speech. This response data may be enhanced with information such as remote NLU data. The response data from the remote speech processing system may be compared to local NLU data to improve a speech processing model on the device. Thus, the device may perform supervised on-device learning based on the remote NLU data. The device may determine differences between the updated speech processing model and an original speech processing model received from the remote system and may send data indicating these differences to the remote system. The remote system may aggregate data received from a plurality of devices and may generate an improved speech processing model.