Hybrid Compute System for Machine Learning Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing applications that use knowledge graphs for query suggestion and composition assistance often rely on remote server-based machine-learning models, leading to increased latency and inefficient use of computing resources due to the need for frequent data communication over networks.
Innovation Solution
Implementing a hybrid approach that utilizes local machine-learning models on client devices for initial processing, with the option to leverage remote resources only when local models fail to meet quality thresholds, thereby reducing network communication and enhancing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If remote server-based machine-learning models are used for query suggestion and composition assistance, then service capability is provided, but user-perceived latency increases and network data transfer is required
Solution Approach 1:
The system segments machine-learning processing into two parts: local execution for simple queries and remote execution for complex queries. The scenario router divides work based on query complexity, enabling local models to handle straightforward tasks independently while remote models handle complex tasks that require additional computational resources, thereby reducing latency for common operations.
Solution Approach 2:
The patent implements local machine-learning models on client devices to handle specific scenarios where local execution is sufficient. By deploying models locally for common query types, the system achieves fast response times without requiring remote server access, directly addressing the latency issue while maintaining service capability for appropriate scenarios.
2Reliability
If remote server-based machine-learning models are used, then service capability is provided, but computing resource efficiency decreases due to frequent network communication
Solution Approach 1:
The system segments processing workload based on query complexity and local capability. Simple queries are handled locally to avoid network round-trips, while complex queries are sent to remote servers. This segmentation optimizes computing resource efficiency by executing processing locally when possible, eliminating unnecessary network communication overhead.
Solution Approach 2:
Client devices perform self-service by executing machine-learning models locally for query processing. This self-service capability allows the system to handle common tasks independently without requiring remote server resources, significantly improving computing resource efficiency by eliminating redundant network communication and server processing for simple queries.
3Loss of time
If local machine-learning models are used, then user-perceived latency is reduced and network data transfer is minimized, but model capability may be insufficient for complex scenarios
Solution Approach 1:
The system dynamically selects between local and remote model execution based on scenario complexity. The scenario router evaluates each query and determines the appropriate execution location in real-time. This dynamic approach allows the system to achieve fast local response times for simple queries while maintaining the ability to access powerful remote models for complex scenarios, thus balancing latency and capability.
Solution Approach 2:
The hybrid system provides multi-functionality by combining local model execution for simple scenarios with remote model execution for complex scenarios. This universal approach enables the system to handle both low-latency requirements and high-capability requirements within a single architecture, making the system adaptable to diverse query types without sacrificing either latency performance or model capability.
Data Source
AI summary
The technology described herein uses local computing resources, rather than remote resources (e.g., server based), to provide a result upon determining that the local resource is capable of providing the result with above a threshold quality. When a local machine-learning model is not capable of providing the result with above the threshold quality, then a remote machine-learning model may be used to provide the result. A goal of the technology is to select the most efficient resource to provide a result without significantly compromising the quality of the result. The technology described herein makes a series of determinations to identify one or more machine-learning model results that may be provided locally with or without hybrid resources. Different machine-learning model results may be provided using different hybrid workflows. In aspects, a remote result and a local result are generated and ranked by the client.