Cloud LLM Selection by Request Classification to Reduce Costs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The high computational resource requirements and associated costs of operating Large Language Models (LLMs) pose a significant challenge, particularly in cloud computing environments, where pricing models are based on usage metrics such as API requests, token counts, or compute time.
Innovation Solution
An AI automation system employs a classification model to select an LLM based on request classification, applies a similarity model to optimize token counts, and utilizes GPU time-slicing and LoRA fine-tuning to reduce costs and improve efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single large language model is used to handle all types of requests, then the system is simple to manage, but the operational costs increase due to unnecessary usage of expensive models for simple tasks
Solution Approach 1:
The patent segments the request processing workload by dividing requests into different categories (simple vs. complex) and routing them to appropriately sized LLM instances. This segmentation allows the system to use smaller, cheaper models for simple tasks and reserve larger, more capable models only for complex requests, thereby reducing overall operational costs while maintaining system manageability through automated routing.
Solution Approach 2:
The patent implements dynamic instance selection where the system automatically chooses the appropriate LLM instance size based on the characteristics of each incoming request. This dynamic approach allows the system to adapt its resource usage in real-time, selecting smaller instances for straightforward tasks and larger instances only when necessary, thus optimizing the balance between management simplicity and cost efficiency.
2Loss of energy
If smaller LLM instances are used to reduce costs, then operational costs decrease, but the ability to handle complex requests deteriorates
Solution Approach 1:
The system dynamically selects LLM instance sizes based on request characteristics, automatically scaling up to larger instances when complex requests are detected and using smaller instances for simple tasks. This dynamic adaptation ensures that cost reductions from using smaller models do not compromise the system's ability to handle complex requests, as the appropriate model size is chosen for each specific request type.
Solution Approach 2:
The patent introduces an intermediary routing mechanism that analyzes incoming requests and directs them to appropriately sized LLM instances. This intermediary layer ensures that complex requests are automatically routed to larger, more capable models while simple requests go to smaller, more cost-effective models, thus preserving complex request handling capability while reducing overall operational costs.
3Reliability
If larger LLM instances are used to improve response quality, then the response quality increases, but the token consumption and costs increase
Solution Approach 1:
The patent segments requests by complexity and routes them to appropriately sized LLM instances. This segmentation ensures that simple requests are handled by smaller models with lower token consumption, while only complex requests that truly require high response quality are routed to larger models. This approach maintains necessary response quality for complex tasks while reducing overall token consumption across the system.
Solution Approach 2:
The system changes the parameter of model instance size based on request characteristics. By adjusting which model size is deployed for each request type, the system optimizes the balance between response quality and token consumption, using smaller models for simple tasks to reduce token usage while reserving larger models for complex requests where high response quality is critical.
4Loss of energy
If multiple LLM instances of different sizes are maintained, then cost optimization is improved, but the device complexity increases
Solution Approach 1:
The patent implements an automated routing system that self-manages the distribution of requests to appropriate LLM instances based on request characteristics. This self-service approach to instance selection automates the complexity of managing multiple instance sizes, allowing the system to maintain multiple LLM instances for cost optimization while the automated routing logic handles the management complexity, reducing the burden on operators.
Data Source
AI summary
Systems or methods for the selection of large language models (LLMs). A system receives a request from a service that hosts an application. The request is configured to be processed by an LLM to generate a response. The system applies a classification model to the request to determine the class of the request. The classification model is a language model trained to receive text and classify the text into a plurality of classes. The system selects an LLM from a plurality of candidate LLMs based in part on the determined class of the request and recommends the selected LLM to the application.