Cooperative On-Device Assistant Models Across Low-Memory Device Groups
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing automated assistant devices face limitations in processing power and memory, leading to less robust and accurate local components, especially in older and less costly devices, which can't execute or store necessary models effectively.
Innovation Solution
Dynamically adapt on-device models and processing roles among a group of assistant devices based on individual capabilities, distributing them cooperatively to enhance robustness and accuracy, reducing the need for remote processing and improving security and latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If more processing components are executed locally at the assistant device, then latency decreases and security improves, but the processing power and memory capacity required increase
Solution Approach 1:
The processing pipeline is segmented into multiple components (wake word detection, ASR, NLU, fulfillment) that can be distributed across different devices. Each device executes only the components it can handle locally, while other components are executed remotely, allowing the system to achieve robust processing without requiring each individual device to have substantial memory capacity.
Solution Approach 2:
The patent transitions from a single-device processing model to a multi-device distributed processing model. By adding the dimension of spatial distribution across multiple devices, the system can execute more processing components locally (across the group) without requiring each individual device to have increased memory capacity.
2Loss of time
If more processing components are executed locally at the assistant device, then latency decreases, but the processing power required increases
Solution Approach 1:
The processing pipeline is divided into discrete components that can be selectively executed. Devices execute only the components they are capable of handling locally (such as wake word detection and basic ASR), while more computationally intensive components (such as complex NLU and fulfillment) are executed remotely, reducing latency for simple operations without overloading device processing power.
Solution Approach 2:
The system performs partial processing locally and partial processing remotely. By executing only the necessary portion of the processing pipeline locally (enough to reduce latency for critical path operations) and completing the remainder remotely, the system achieves latency reduction without requiring full processing power locally.
3Ease of manufacture
If older and less costly assistant devices are used, then device cost decreases, but the robustness and accuracy of local components decrease
Solution Approach 1:
Multiple devices are merged into a cooperative group where their capabilities are combined. Older or less capable devices can participate in the distributed processing pipeline without needing to individually possess high accuracy components, as the collective accuracy of the group compensates for individual limitations.
Solution Approach 2:
The patent introduces a coordination mechanism that acts as an intermediary between devices with limited local processing capabilities and the remote processing infrastructure. This intermediary manages the distribution of processing tasks, allowing低成本 devices to achieve accurate results through coordinated remote execution of components they lack.
Data Source
Figure 1A
Figure 1B1
Figure 1B2
AI summary
Implementations are directed to dynamically adapting which assistant on-device model(s) are locally stored at assistant devices of an assistant device group and/or dynamically adapting the assistant processing role(s) of the assistant device(s) of the assistant device group. In some of those implementations, the corresponding on-device model(s) and/or corresponding processing role(s), for each of the assistant devices of the group, is determined based on collectively considering individual processing capabilities of the assistant devices of the group. Implementations are additionally or alternatively directed to cooperatively utilizing assistant devices of a group, and their associated post-adaptation on-device model(s) and/or post¬ adaptation processing role(s), in cooperatively processing assistant requests that are directed to any one of the assistant devices of the group.