LLM Batch Processing with Greedy Request Drop Thresholds
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing batch processing methods for large language models (LLMs) are inefficient due to unpredictable response sizes and execution times, leading to resource wastage and suboptimal hardware utilization, particularly in decoder-based transformer architectures.
Innovation Solution
Implementing a greedy request drop (GRD) functionality that stops and re-queues requests within a batch when a threshold is reached, optimizing hardware resource utilization by addressing greedy requests that exceed a defined execution time or output token count.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If batch processing is used to process multiple requests simultaneously, then hardware resource utilization should improve, but unpredictable response sizes and execution times cause resource wastage and suboptimal hardware utilization
Solution Approach 1:
The patent implements dynamic batch processing by introducing a request drop threshold that dynamically adjusts during batch execution. When requests are completed or dropped, the threshold is updated to allow flexible stopping points, enabling the system to adapt to variable response sizes and execution times in real-time, thereby optimizing hardware resource utilization while reducing resource wastage.
Solution Approach 2:
The patent changes the parameter of request processing by introducing a request drop threshold that can be modified during batch execution. This parameter change allows the system to stop processing a batch when certain conditions are met, accommodating unpredictable response sizes and execution times while maintaining efficient hardware resource utilization and minimizing resource wastage.
2Reliability
If batch processing continues until all requests are completed, then processing completeness is ensured, but overall latency increases due to greedy requests exceeding execution time threshold
Solution Approach 1:
The patent segments the batch processing into manageable portions by introducing a request drop threshold. When requests are completed or dropped, the batch is divided and reprocessed in subsequent batches. This segmentation ensures that processing completeness is maintained through multiple passes while reducing overall latency by avoiding the delay of waiting for all requests in a single batch to complete.
Solution Approach 2:
The patent implements periodic batch processing where requests are processed in multiple cycles. The request drop threshold is reset and adjusted after each batch, allowing the system to periodically reprocess requests that were dropped. This periodic action ensures processing completeness over time while significantly reducing overall latency compared to waiting for all requests to complete in a single batch.
3Loss of energy
If the request drop threshold is set low, then resource wastage is reduced, but more requests need to be re-processed increasing processing time
Solution Approach 1:
The patent implements feedback mechanisms where the request drop threshold is adjusted based on actual request completion and drop patterns. The system monitors which requests are dropped and why, using this feedback to optimize the threshold setting. This feedback loop reduces resource wastage by preventing unnecessary reprocessing while minimizing processing time by avoiding overly aggressive threshold settings that would require excessive reprocessing.
Data Source
AI summary
An apparatus comprises at least one processing device configured to execute, in a first batch, a set of requests utilizing at least one machine learning model, to determine, during execution of the first batch, whether any request drop activation conditions are triggered and to establish a request drop threshold responsive to determining that at least one request drop activation condition has been triggered. The at least one processing device is also configured to determine whether any of the requests are still executing when the request drop threshold is reached, to stop execution of the first batch responsive to determining that at least a subset of the first set of requests are still executing when the request drop threshold is reached, and to execute at least one request in the subset of the first set of requests in a second batch utilizing the at least one machine learning model.