Hybrid LLM Execution Across Compute Instances Under Data Constraints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing optimization methods for large language model (LLM) inference do not consider data-specific constraints, leading to inefficiencies in computational resource utilization and regulatory compliance.
Innovation Solution
A computer-implemented method for managing LLM execution on distributed compute instances by considering task-specific data sensitivity and complexity information, along with compute instance capabilities, to select optimal instances for parallel task execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If workload-aware allocation groups queries with similar sequence lengths into micro-batches to enhance efficiency, then computational resource utilization is improved, but data-specific constraints and regulatory requirements are not considered
Solution Approach 1:
The patent segments the query allocation process into multiple dimensions: computational characteristics (sequence length, model type) and data characteristics (sensitivity levels, regulatory requirements). This segmentation allows independent optimization of each dimension while ensuring compliance constraints are met before resource allocation decisions are made.
Solution Approach 2:
The patent applies local quality by creating specialized allocation strategies for different data sensitivity levels and regulatory contexts. High-sensitivity data queries are routed to compute instances with appropriate security clearances and compliance certifications, while less sensitive queries can utilize more general-purpose resources, optimizing both compliance and efficiency.
2Ease of manufacture
If Round-Robin Allocation assigns queries to workers in cyclic order, then implementation simplicity is maintained, but load imbalance occurs due to ignoring computational demand
Solution Approach 1:
The patent transitions from static Round-Robin allocation to dynamic allocation that adapts to both computational characteristics and data characteristics of queries. The system continuously evaluates query requirements and compute instance availability, adjusting allocation decisions in real-time to maintain load balance while considering data sensitivity and regulatory constraints.
3Productivity
If iteration-level scheduling dynamically replaces completed sequences to maintain consistent batch sizes, then computational resource utilization is improved, but data-specific limitations are not considered
Solution Approach 1:
The patent introduces data characteristic parameters (sensitivity levels, regulatory requirements) alongside computational parameters (sequence length, batch size) in the scheduling decision process. This multi-parameter approach allows the system to maintain consistent batch sizes for efficiency while simultaneously respecting data-specific constraints through parameter-based routing and filtering.
Data Source
AI summary
The invention relates to a computer-implemented method for managing execution of a task on a trained machine learning model configured on one or more compute instances in a distributed computing network. The invention also relates to an apparatus comprising means for carrying out the method and a computer program comprising instructions which, when executed by a processor, cause the processor to perform the method. The method comprises obtaining task-specific information about the task, including data sensitivity information of the task and task complexity information, obtaining one or more compute instance characteristics associated with each of the one or more compute instances, the compute instance characteristics including a data sensitivity capability and a task complexity capability of the compute instance, selecting a compute instance of the one or more compute instances for executing the task based on the task-specific information about the task and on the one or more compute instance characteristics, and executing at least part of the task on the selected compute instance.