Hybrid RAG Query Decomposition for Low-Latency Mobile AI
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Mobile devices with limited computational resources face challenges in performing complex tasks due to high latency when accessing cloud-based systems for retrieval augmented generation, and on-device models may struggle with complex queries, leading to inefficient performance.
Innovation Solution
A hybrid AI approach that decomposes input queries into sequential and parallel sub-queries, optimizing execution plans to minimize latency and maximize accuracy by leveraging both local and cloud-based resources effectively.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If mobile devices access cloud-based systems for retrieval augmented generation, then task performance capability is improved, but latency increases
Solution Approach 1:
The patent segments the query processing into multiple sub-queries that can be executed in parallel. The input prompt is decomposed into several independent sub-prompts that are processed simultaneously by different language models, reducing the overall time required compared to sequential processing or single cloud-based execution.
Solution Approach 2:
The patent introduces a hybrid architecture that operates across multiple dimensions - combining on-device execution with selective cloud-based retrieval. This multi-dimensional approach allows the system to leverage both local instant response capabilities and remote comprehensive processing power, optimizing the balance between latency and task performance capability.
2Loss of time
If mobile devices use on-device models, then latency is reduced, but task performance capability deteriorates
Solution Approach 1:
The patent segments complex tasks into simpler sub-tasks that can be handled by on-device models. By breaking down complex queries into multiple sub-prompts, the system enables local models to process portions of the task independently, reducing latency while maintaining adequate performance for each sub-task.
Solution Approach 2:
The patent applies partial action by using on-device models for immediate response generation and supplementing with cloud-based retrieval only when necessary. This partial use of cloud resources minimizes latency while achieving sufficient task performance for many common queries without requiring full cloud-based processing.
3Measurement precision
If complex queries are processed sequentially, then accuracy is improved, but latency increases
Solution Approach 1:
The patent segments complex queries into multiple independent sub-queries that can be processed in parallel. Each sub-prompt is evaluated for sequential dependency, and independent sub-prompts are executed simultaneously by different language models, maintaining accuracy through comprehensive processing while reducing overall latency through parallel execution.
Solution Approach 2:
The patent implements dynamic query processing by evaluating sequential dependencies among sub-prompts and adapting the execution plan accordingly. The system dynamically determines which sub-prompts can be processed in parallel and which require sequential execution, optimizing the balance between accuracy and latency based on the specific query structure.
Data Source
AI summary
Certain aspects of the present disclosure provide techniques and apparatus for improved machine learning. In an example method, an input prompt for machine learning is received, and the input prompt is decomposed to generate a set of sub-prompts. A sequence of requests for sub-prompts of the set of sub-prompts that have sequential dependency is generated, and a parallel request for sub-prompts of the set of sub-prompts that do not have sequential dependency is generated. Based on evaluating the sequence of requests and the parallel request, an execution plan for using one or more machine learning models to generate a response to the input prompt is generated. The response to the input prompt is output according to the execution plan.