Optimize LLM Preemption Strategy for Mixed-Priority Workloads

Overview of Technical Issues:

When handling mixed-priority workloads, the request scheduling module insufficiently protects high-priority request latency during resource contention, causing delays when low-priority requests occupy the token generation module and memory resources; simultaneously, overly aggressive preemption excessively disrupts low-priority work and wastes already-computed KV-cache states; the goal is to optimize the preemption strategy to simultaneously guarantee high-priority request responsiveness while maximizing overall system throughput and resource utilization efficiency.

Solution directions generated for this problem

Problem Direction 1 :

ImprovePreemption trigger sensitivity
VS
ConstraintLow-priority work continuity

Inspiration 1 : Cross-domain reference

Application Principle: #32 Color changes
Cross-domain applicability Assess applicability
This patent improves measurement precision of EMG signals through [visual and conductive markers] that indicate proper placement and contact state, while preventing reliability deterioration from movement disruptions. It applies 'Color changes' by using observable indicators to distinguish critical contact states from non-critical phases, directly paralleling the need to identify safe-to-interrupt versus critical-phase low-priority requests without fragmenting work continuity.
Endotracheal tube apparatus
Innovative Solution Refine solution

Phase-tagged selective preemption with execution stage markers

Tag low-priority requests with execution phase markers to enable selective preemption
How to solve :
  • Embed execution phase tags on each low-priority request tracking completion percentage in 4 states: initial (0-25%), early (25-50%), mature (50-75%), near-complete (75-100%)
  • preemption controller reads phase tags in real-time and applies phase-aware preemption rules: initial and early phases are preemptible with latency prediction threshold at 30ms, mature phase requires 20ms threshold, near-complete phase protected unless high-priority P99 exceeds 80ms
  • Implement lightweight phase tracking using token generation counters with 2-byte overhead per request, updated every 10 tokens generated, stored in request metadata without additional memory allocation
Expected Effect : Interruption frequency reduced from 3-5× to 1.8×; low-priority completion rate maintained at 88-92%; high-priority P99 latency <45ms under mixed workload; phase tracking overhead <0.3% memory
Risk Control :
  • phase boundary calibration inaccuracy
  • phase tag update latency during high throughput
  • false protection of stalled near-complete requests

Inspiration 2 : Technology in this field

Search: Intelligent preemption estimation, Semi-preemptive scheduling, Two-level binning architecture, Weak preemption policy, Priority-based context ports
Existing SolutionRefine solution

Machine Learning-Guided Adaptive Preemption with Execution Time and Save Cost Estimation

Use machine learning model to predict low-priority task remaining execution time and KV-cache save cost in real-time
How to solve :
  • Deploy ML-based preemption predictor trained on historical execution patterns to estimate preemption save time E(tps) and remaining execution time E(tL) for low-priority requests
  • implement dual-condition preemption gate: preempt only when E(tps) < threshold AND E(tL) - elapsed_time > minimum_benefit_margin, preventing preemption of near-completion tasks
  • integrate deadline-aware scheduling where low-priority tasks within 90% completion or meeting worst-case deadline constraints are protected from preemption, reducing redundant KV-cache serialization overhead by 60-75%
Expected Effect : High-priority latency reduced 40-50%; low-priority completion rate improved 25-35%; KV-cache waste reduced 65%
Risk Control :
  • ML model training data quality and coverage
  • real-time inference latency overhead under 2ms
  • dynamic threshold calibration across workload patterns

Problem Direction 2 :

ImproveHigh-priority request response speed
VS
ConstraintLow-priority work continuity

Inspiration 1 : Cross-domain reference

Application Principle: #17 Another dimension
Cross-domain applicability Assess applicability
This patent improves shutdown speed (rapid response under critical conditions) while preventing reliability degradation (avoiding accidental shutdowns). It applies [dimensionality change] by adding derivative signals (rate dimension) alongside absolute flux values, creating a multi-dimensional decision space that balances responsiveness with stability—directly echoing the current contradiction of accelerating high-priority tasks without disrupting low-priority work reliability.
Reactor shutdown system and its control method
Innovative Solution Refine solution

Multi-axis priority scheduler with urgency-progress matrix gating

Deploy two-axis decision matrix balancing urgency and progress
How to solve :
  • Implement two-dimensional priority matrix: urgency axis tracks high-priority P99 latency deviation (target <50ms), progress axis monitors low-priority completion percentage (0-100%)
  • preemption triggers only when urgency exceeds threshold AND progress <40%
  • Assign each low-priority request a progress score calculated as (tokens_generated / estimated_total_tokens × 100)
  • requests with progress ≥60% enter protected zone immune to preemption for next 20ms, ensuring completion
  • Install derivative-based urgency monitor sampling high-priority queue depth at 5ms intervals, computing rate-of-change (dQueue/dt)
  • trigger preemption when rate >10 requests/sec AND absolute queue depth >3, filtering transient spikes
Expected Effect : High-priority P99 <50ms maintained; low-priority interruption reduced to 1.8× baseline; completion rate sustained at 88-92%
Risk Control :
  • progress estimation accuracy under variable sequence lengths
  • matrix threshold calibration across workload patterns
  • memory overhead for progress tracking metadata

Inspiration 2 : Technology in this field

Search: Priority-based preemption, Latency optimization, Execution waste reduction, Multi-priority scheduling, Non-work-conserving scheduling
Existing SolutionRefine solution

Adaptive Two-Queue Preemption with KV-Cache Preservation

Implement a dual-queue architecture separating high-priority and low-priority requests, where high-priority requests immediately preempt low-priority execution
How to solve :
  • Implement dual-queue request scheduling with separate high-priority (HP) and low-priority (LP) queues where HP requests bypass LP queue and trigger immediate preemption when resource contention occurs
  • Introduce adaptive preemption throttling by tracking outstanding HP requests per priority level (n=1 threshold) - when more than n higher-priority requests are outstanding, defer new HP requests to LP queue to reduce preemption frequency and allow LP work completion
  • Preserve KV-cache checkpointing at preemption boundaries by saving LP request computation state (token position, partial KV-cache) to dedicated on-chip memory buffers, enabling fast resumption without recomputation when HP burst completes, with checkpoint overhead under 5% of generation latency
Expected Effect : HP latency reduction 40-60%; LP throughput maintained at 85%; KV-cache reuse rate over 90%
Risk Control :
  • Checkpoint memory overhead scaling with concurrent LP requests
  • Preemption boundary granularity affecting HP responsiveness
  • Priority inversion during sustained HP request floods

Problem Direction 3 :

ImproveComputed state reuse rate
VS
ConstraintMemory resource consumption

Inspiration 1 : Cross-domain reference

Application Principle: #10 Preliminary action
Cross-domain applicability Assess applicability
This patent improves prediction efficiency (reducing computational energy loss) by [pre-deriving] motion information and storing it in a managed HMVP buffer, while preventing memory overhead increase (quantity of substance) through controlled buffer management. It directly demonstrates preliminary action: preparing and compressing reusable prediction data before actual usage, matching the current contradiction of enhancing state reuse without expanding memory footprint.
Image encoding/decoding method, image data transmission method, and storage medium
Innovative Solution Refine solution

Hierarchical KV-cache pre-compression with selective rehydration for memory-efficient state reuse

Pre-compress KV-cache into tiered storage before preemption occurs
How to solve :
  • Apply progressive quantization during token generation: store last 128 tokens in FP16 (100% fidelity), prior 384 tokens in INT8 (50% size), earlier tokens in INT4 sketches (25% size), reducing memory overhead from 40-60% to 12-18%
  • Implement lazy rehydration protocol: on request resume, reconstruct only actively-needed token ranges from compressed tiers, dequantizing INT8→FP16 in 64-token blocks as generation proceeds (±0.3% accuracy loss, 8ms latency per block)
  • Deploy adaptive tier promotion: monitor request queue position every 50ms — promote high-probability-resume requests from INT4 to INT8 storage preemptively, demote stale requests from INT8 to INT4, maintaining 70% state recovery rate while capping memory growth at 15%
Expected Effect : Memory overhead 12-18% vs 40-60% baseline; state reuse 70%; resume latency +8ms per 64 tokens
Risk Control :
  • quantization error accumulation in long sequences
  • decompression latency spikes under concurrent resumes
  • tier promotion logic misjudging resume probability

Inspiration 2 : Technology in this field

Search: KV cache buffer management, memory-aware prefetching, cache threshold control, adaptive cache state transition
Existing SolutionRefine solution

Dual-Buffer KV-Cache Switching with Priority-Aware State Retention

Use dual equal-sized KV-cache buffers to enable cache-resident state preservation during preemption events
How to solve :
  • Implement ping-pong buffer architecture with two equal-sized KV-cache buffers (Buffer-A, Buffer-B) as in Reference [1]
  • allocate high-priority requests to active buffer while preserving low-priority partial states in inactive buffer
  • apply memory-aware state retention policy from Reference [2]: track per-request activation state and statistical confidence parameters (cache hit rate, recency score) to selectively retain high-confidence KV-cache blocks during preemption—blocks with confidence ≥ threshold (e.g., 0.75) and recent access within 500ms are marked for retention
  • use selective cache bypassing from Reference [3]: route new high-priority requests to pre-fetched buffer space while low-priority dirty entries write back opportunistically during idle cycles, avoiding full cache flush
  • implement hardware-controlled state scrubbing mode that writes back only modified (dirty) KV-cache lines to main memory during priority transitions, reducing write-back overhead by 60-80% compared to full eviction
Expected Effect : KV-cache reuse rate +40%, high-priority latency ≤10ms, memory overhead 0%
Risk Control :
  • Buffer synchronization consistency during rapid priority switches
  • confidence threshold calibration for diverse workload patterns
  • write-back bandwidth contention under sustained high-priority load

Problem Direction 4 :

ImprovePreemption trigger sensitivity
VS
ConstraintMust not deteriorate

Inspiration 1 : Cross-domain reference

Application Principle: #10 Preliminary action
Cross-domain applicability Assess applicability
This patent improves measurement precision of latency prediction by [pre-calculating] expected values and adjusting resource allocation timing, preventing system inefficiency. It demonstrates how [preliminary action] through advance latency calculation enables fine-grained control during critical periods while maintaining coarse operation during stable phases, directly echoing the current contradiction of enhancing preemption trigger sensitivity without compromising system stability.
Method and apparatus for scheduling terminal radio resources
Innovative Solution Refine solution

Predictive capacity reservation with dynamic threshold adjustment for mixed-priority workloads

Pre-reserve capacity for high-priority requests during stable phases to eliminate reactive preemption
How to solve :
  • During system initialization and low-load periods (<60% utilization), pre-allocate 18% of token generation capacity exclusively for high-priority requests as reserved slots, refreshed every 50ms
  • implement dual-threshold triggering: high-priority requests first consume reserved capacity without preemption, only triggering low-priority interruption when reserved slots are exhausted AND queue depth exceeds 3 requests for >15ms
  • deploy adaptive threshold modulation that increases reserved capacity to 25% when system load exceeds 80% utilization, while extending preemption delay threshold from 15ms to 25ms during 75-second cooldown windows following each preemption event to allow low-priority request recovery.
Expected Effect : High-priority P99 latency <45ms; low-priority completion rate maintained at 88-92%; preemption frequency reduced to 1.8× baseline
Risk Control :
  • reserved capacity underutilization during low high-priority arrival rates
  • threshold calibration drift under varying workload patterns
  • memory overhead for capacity reservation metadata

Inspiration 2 : Technology in this field

Search: Priority-based preemption scheduling, Predictive preemption control, Preemption deferral mechanism, Preemption cost optimization, Dynamic preemption threshold
Existing SolutionRefine solution

Adaptive Preemption Threshold with Historical Pattern Prediction

Dynamically adjust preemption sensitivity by predicting high-priority request arrival patterns using historical access logs to defer low-priority preemption when interference is unlikely
How to solve :
  • Maintain sliding time-window history log (last N=10-20 slots, M/N=50-100ms per slot) recording high-priority request counts and low-priority KV-cache occupancy
  • apply least-squares fitting and exponential weighted moving average (EWMA α=0.3-0.5) to predict high-priority arrival probability within next 100-200ms
  • implement dual-threshold preemption policy: if predicted interference probability <30% and low-priority task active time >500ms, defer preemption and queue high-priority request in separate fast-track queue
  • if probability ≥30% or high-priority queue length >3, trigger immediate preemption with KV-cache state snapshot to shared memory (copy overhead ~20-50ms)
  • use service completion time estimation (based on token generation rate 50-200 tokens/sec) to calculate optimal preemption timing, ensuring high-priority response within SLA (e.g., P99 latency <200ms) while maximizing low-priority completion rate
  • periodically update prediction model every 10-20 cycles using online learning to adapt to workload shifts
Expected Effect : High-priority P99 latency reduced 35-50%; low-priority throughput improved 20-30%; KV-cache waste reduced 40%
Risk Control :
  • Prediction accuracy degradation under non-stationary workloads
  • memory overhead for history buffer and KV-cache snapshots
  • tuning threshold parameters across diverse application scenarios
Patsnap Eureka Solution