Cloud Assistant Distillation for Local Query Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud-based voice-enabled assistant services face issues with network bandwidth consumption, increased latency, and reduced privacy due to the need to transfer audio data from user devices for processing, which can be addressed by distilling models to execute locally on client devices.
Innovation Solution
A method to generate distilled models of cloud-based assistant services on client devices by analyzing query patterns and attributes, allowing local processing of user queries, thereby reducing the need for network transmission and enhancing privacy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If cloud-based assistant service processes all queries, then query processing capability is improved, but network bandwidth consumption increases and latency increases
Solution Approach 1:
The patent segments query processing into two parts: a distilled model executed locally on the client device for common query types, and a full cloud-based model for complex queries. This segmentation allows frequent, simple queries to be processed locally without network transmission, reducing bandwidth consumption while maintaining overall processing capability.
Solution Approach 2:
The patent implements local quality by deploying a distilled model specifically optimized for execution on client devices with limited resources. This local model handles the majority of routine queries, while the full-capacity cloud model handles complex queries, creating a quality gradient that matches resource availability.
2Productivity
If cloud-based assistant service processes all queries, then query processing capability is improved, but latency increases
Solution Approach 1:
The patent segments query processing to identify and handle different query types differently. Simple, frequent queries are routed to the local distilled model for immediate processing, eliminating network transmission delays. Complex queries are routed to the cloud, maintaining processing capability while minimizing latency for the majority of interactions.
Solution Approach 2:
The patent performs preliminary action by pre-training a distilled model on the client device that captures the essential capabilities of the full cloud model. This preliminary model is ready to immediately process common queries without waiting for cloud communication, reducing latency for routine operations.
3Productivity
If cloud-based assistant service processes all queries, then processing capabilities are improved, but user privacy is reduced
Solution Approach 1:
The patent segments query handling to process sensitive or routine queries locally on the client device using the distilled model, avoiding network transmission of audio data. Only queries that exceed the local model's capabilities are transmitted to the cloud, minimizing privacy exposure while maintaining processing power for complex tasks.
Solution Approach 2:
The patent implements self-service by enabling the client device to independently process a significant portion of queries using the locally deployed distilled model. This self-service capability handles routine interactions without requiring cloud communication, thereby preserving user privacy while maintaining functional capability.
Data Source
AI summary
A method includes receiving user queries directed toward a cloud-based assistant service. For each received user query directed toward the cloud-based assistant service, the method also includes extracting one or more attributes from the user query and logging the user query into one or more of a plurality of category buckets based on the one or more attributes extracted from the user query. The method also includes determining when at least one of the plurality of category buckets includes a threshold number of the user queries logged into the at least one category bucket, and when the at least one of the plurality of category buckets includes the threshold number of the user queries, generating a distilled model of the cloud-based assistant service. The distilled model of the cloud-based assistant service is configured to execute on one or more target client devices.