On-Device LLM Agent Routing Across Expert Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AI models face limitations in on-device environments due to high computational costs and resource inefficiencies when processing large datasets, particularly with the MoE architecture, which requires high-level VRAM and is challenging to quickly adapt to varying contexts.
Innovation Solution
A method and system for providing an on-device specialized AI agent that optimizes application models using a mixture of experts (MoE) architecture, allowing efficient decision-making based on a large language model (LLM) to process tasks efficiently across different domains, utilizing a master model and secondary models like sLLM, MoELM, and SMs to manage resource allocation and task distribution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a single large AI model is used to process various tasks, then comprehensive task coverage is achieved, but computational cost and processing time increase significantly
Solution Approach 1:
The patent divides a large AI model into multiple smaller expert models, each specialized in processing specific types of tasks. The gating network segments incoming tasks and routes them to appropriate expert models, achieving both comprehensive task coverage and efficient processing by activating only the necessary subset of models for each task.
Solution Approach 2:
The system dynamically selects which expert models to activate based on the specific task requirements. The gating network evaluates each incoming task and determines the optimal combination of expert models to process it, allowing the system to adapt its computational resources dynamically rather than activating all models for every task.
2Productivity
If MoE architecture is used to activate only specific portions of the model, then computational efficiency is improved, but high-level VRAM and fine-tuning resources are required
Solution Approach 1:
The patent segments the large model into smaller expert models that can be stored and executed more efficiently in on-device memory. By dividing the model into specialized components, each expert model requires less VRAM to operate, making the system feasible for on-device deployment while maintaining computational efficiency.
Solution Approach 2:
The gating network acts as an intermediary that manages memory allocation and model selection. It coordinates between the input task and the appropriate expert models, optimizing memory usage by loading only the necessary expert models into VRAM based on task requirements, thereby reducing overall VRAM requirements.
3Reliability
If traditional AI models are deployed in on-device environments, then data privacy is maintained, but computational resources are insufficient for processing large datasets
Solution Approach 1:
The patent segments the AI model into compact expert models that can be deployed within limited on-device computational resources. Each expert model is specialized and optimized for specific task types, allowing the system to process diverse data locally without requiring excessive computational power, thus maintaining data privacy while enabling on-device processing.
4Measurement precision
If domain-specific AI models are created for different tasks, then processing accuracy for specific domains is improved, but model selection and system complexity increase
Solution Approach 1:
The gating network serves as a universal task router that handles multiple types of tasks by evaluating task characteristics and selecting appropriate expert models. This single multi-functional component manages the complexity of coordinating multiple domain-specific expert models, allowing the system to maintain high domain processing accuracy without proportionally increasing overall system complexity.
Data Source
AI summary
A method for providing an artificial intelligence (AI) agent based on a large language model (LLM) applying an artificial intelligence model including a plurality of models includes: executing an on-device AI agent service; acquiring predetermined input data based on the executed on-device AI agent service; determining a domain according to the acquired input data; deciding an application model, which is an AI model that will process a task according to the determined domain; generating output data for the input data based on the decided application model; and providing the generated output data based on the on-device AI agent service.