Optimize LLM Token Throughput for Multi-Tenant Deployment
Overview of Technical Issues:
In multi-tenant LLM deployment, the memory management unit's context switching operations between tenants create a harmful effect by consuming computational cycles and memory bandwidth without generating tokens, while simultaneously the token processing unit provides insufficient throughput per tenant due to resource fragmentation and reduced batch efficiency when handling requests with varying sequence lengths and arrival patterns across multiple tenants; the goal is to maximize aggregate token throughput across all tenants while maintaining acceptable per-tenant latency.
Solution directions generated for this problem
Problem Direction 1 :
ImproveContext switching overhead
VSConstraintMemory allocation per tenant
Inspiration 1 : Cross-domain reference
Application Principle: #10 Preliminary action
Cross-domain applicability
This patent improves loss of time (reduces latency and processing cycles) by using preliminary action—pre-calculating transformation parameters and memory addresses in specialized hardware—while avoiding increased quantity of substance (minimizes memory storage requirements through on-chip processing). It demonstrates how pre-computation and local processing eliminate runtime overhead without expanding resource footprint, directly echoing the current contradiction of reducing context switching time while constraining per-tenant memory allocation.
Image transformation for machine learning
Innovative Solution Refine solution
Asynchronous dual-buffer context pre-staging with compute-phase memory transfer
Pre-stage next tenant context during current tenant compute phase
How to solve :
- Allocate dual 2GB staging buffers per GPU — while tenant A processes tokens (compute-bound, 70-80% memory idle), DMA controller pre-loads tenant B's KV cache into buffer 1 using idle bandwidth
- switching becomes atomic pointer swap (12-18 CPU cycles, <0.3% overhead) instead of full memory transfer
- rotate buffers across tenant queue for continuous pre-staging
- Implement compute-phase bandwidth monitor — hardware profiler tracks real-time memory utilization during matrix multiplication
- triggers pre-fetch when bandwidth drops below 25% for ≥8ms, ensuring zero interference with active processing
- abort and retry if utilization exceeds 30%
- Deploy compressed context format (4:1 ratio) in staging buffers — decompress on-the-fly during pointer swap using dedicated hardware decoder (latency 2-3ms, overlapped with context activation)
- effective per-tenant footprint remains 3.2GB (2GB active + 0.5GB×2 staging + 0.2GB metadata) supporting 25+ tenants on 80GB GPU
Expected Effect : Switching overhead 2.8-4.2%; tenant capacity maintained at 24-26 per GPU; memory bandwidth utilization 91-94%
Risk Control :
- compute-phase prediction accuracy insufficient causing pre-fetch conflicts
- decompression latency variance under heterogeneous sequence lengths
- staging buffer pointer synchronization failure during high-frequency switches
Inspiration 2 : Technology in this field
Search: Dynamic memory-mapping overallocation, Hardware context switching, Cache state preservation, Compiler-assisted threading, Dynamic pipeline partitioning
Existing SolutionRefine solution
TLB-Aware Dynamic Memory Mapping with Overallocation for Multi-Tenant Context Switching
Optimize memory layouts to minimize TLB eviction events by intentionally reserving unused address ranges to maximize TLB utilization
How to solve :
- Establish upper bounds for dynamic structures per tenant (2-4GB limit)
- create arrays of related MMU mappings grouped by tenant, placing structures within arrays while skipping elements that would cause TLB overcommitment
- scan for array-start addresses where entire arrays can be placed without TLB contention, enabling all non-skipped elements to be accessed without TLB replacements
- use fixed upper bounds on data structure sizes determined at compile-time for non-modifiable embedded environments
- implement lookup tables mapping array indexes to actual addresses for sparse allocation
- allow simultaneous access to all tenant memory without flushing memory-management structures
Expected Effect : Context switching overhead reduced to <5% of cycles; memory allocation maintained at 2-4GB per tenant
Risk Control :
- TLB slice distribution uniformity across tenants
- memory fragmentation from reserved address ranges
- lookup table access latency
Problem Direction 2 :
ImproveBatch processing efficiency
VSConstraintSystem scheduling complexity
Inspiration 1 : Cross-domain reference
Application Principle: #1 Segmentation
Cross-domain applicability
This patent applies [Segmentation] by dividing thermal processing into independently controllable receptacle holders, improving throughput (Productivity) while avoiding complex synchronized control systems (Device complexity). Each holder operates asynchronously, enabling parallel processing without global coordination overhead, directly echoing the current contradiction of enhancing batch efficiency while minimizing scheduling complexity.
Apparatus for applying thermal energy to a receptacle and detecting an emission signal from the receptacle
Innovative Solution Refine solution
Length-stratified independent queue batching system with fixed-size execution windows
Partition requests into independent queues by sequence length ranges to eliminate cross-length optimization
How to solve :
- Establish three independent queues for short (128-512 tokens), medium (512-2048 tokens), and long (>2048 tokens) sequences with dedicated GPU memory partitions of 30%, 45%, and 25% respectively
- Each queue executes fixed-window batching: accumulate requests until total tokens reach 85-92% of partition capacity or 50ms timeout, then launch batch without inter-queue coordination
- Implement per-queue token counters with hardware-accelerated summation circuits (latency <0.1ms) to trigger batch execution when cumulative token count crosses threshold, eliminating complex scheduling algorithms
Expected Effect : Batch efficiency 81-86%, scheduler overhead <0.8%
Risk Control :
- queue imbalance under skewed workload
- fixed partition rigidity
- threshold calibration sensitivity
Inspiration 2 : Technology in this field
Search: Heterogeneous batch scheduling algorithms, Adaptive batch sizing optimization, Dynamic job multiplexing control, Sequence length grouping strategy, Makespan minimization scheduling
Existing SolutionRefine solution
Dynamic Batch Size Self-Adjustment with Dual-Layer Job Encoding for Multi-Tenant LLM Scheduling
Implement continual batch processing where each new batch self-adjusts size based on data collected during preceding batch execution to match heterogeneous workload patterns
How to solve :
- Adopt double-layer encoding scheme where first layer encodes tenant request sequences by condition (sequence length ranges) into sub-layers, second layer encodes dynamic batch sizes (Bmin to Bmax) determined by amount of data accumulated since previous batch completion
- processing efficiency increases with batch size as file open/close operations reduce proportionally—processing 10GB as one batch requires 50% fewer operations than two 5GB batches
- implement genetic algorithm-based scheduler with fitness function as reciprocal of makespan, using two-point crossover for first layer and single-point mutation for second layer, population size 200, iterations 400, crossover probability 0.8, mutation probability 0.05
- execute simulation-based job mapping using Least Execution Time technique to map each tenant request to server with minimum simulated completion time considering active jobs, updating server state after each allocation
Expected Effect : Batch efficiency >85% across heterogeneous sequences; scheduling overhead <3% of cycles; aggregate throughput increase 25-35%
Risk Control :
- Genetic algorithm convergence time under high arrival rate variance
- accuracy of execution time prediction for memory-bandwidth-intensive requests
- overhead of maintaining dual-layer encoding structures
Problem Direction 3 :
ImproveMemory bandwidth utilization efficiency
VSConstraintMemory allocation per tenant
Inspiration 1 : Cross-domain reference
Application Principle: #2 Taking out
Cross-domain applicability
This patent improves bandwidth efficiency (Loss of energy) by [extracting] control signaling to shared subcarriers with spreading codes, avoiding dedicated per-user resource allocation (Quantity of substance). It demonstrates how [extraction] of overhead operations from main data paths optimizes resource utilization without proportional capacity increases, directly paralleling the current contradiction of maximizing memory bandwidth for productive work while constraining per-tenant memory allocation.
Method for transmitting control information on the forward link of a wireless network
Innovative Solution Refine solution
Dedicated DMA-based context switching channel with asynchronous pre-staging for multi-tenant LLM memory management
Isolate tenant context switching to dedicated DMA channel separate from token processing memory path
How to solve :
- Deploy dedicated DMA engine with independent 32GB/s bandwidth channel exclusively for tenant context transfers, physically separated from main 256GB/s token processing bus
- Implement asynchronous pre-staging protocol where next tenant's 3.2GB KV cache loads into staging buffer during current tenant's compute phase (GPU utilization >70%), achieving zero-bandwidth switching
- Configure dual-buffer architecture with 3.2GB active buffer + 3.2GB staging buffer per tenant, pointer swap completes in <50μs versus 8-12ms full transfer, maintaining 3.5GB average allocation
Expected Effect : Productive bandwidth utilization 91-93%; switching overhead <3%; tenant capacity maintained at 2-4GB effective allocation
Risk Control :
- DMA channel contention during simultaneous tenant arrivals
- staging buffer synchronization failures
- pointer swap race conditions
Inspiration 2 : Technology in this field
Search: Dynamic Bandwidth Allocation, Multi-Tenant Memory Management, Bandwidth-Aware Scheduling, Memory Bandwidth Optimization, Transaction Limit Control
Existing SolutionRefine solution
Dynamic Memory Bandwidth Budget Allocation with Shared Credit Pool for Multi-Tenant LLM Serving
Implement shared credit pool architecture where each tenant receives reserved private memory bandwidth credits plus access to shared credits for dynamic allocation
How to solve :
- Establish dynamic memory bandwidth budget system with private credits (2-4GB per tenant baseline) and shared credit pool (256+ credits) managed by T-Limit manager
- each tenant receives minimum reserved bandwidth allocation ensuring guaranteed service level, while unused bandwidth is redistributed to active tenants via picker logic based on priority ranking and current utilization
- Implement bandwidth-aware scheduling that monitors real-time memory bandwidth consumption per tenant using counters tracking outstanding transactions, adjusting throttle controls iteratively through feedback loop when bandwidth utilization approaches capacity thresholds
- Deploy work-conserving allocation policy where spare capacity (total_capacity - Σmin(budget_t, demand_t)) is distributed proportionally to tenants with demand exceeding budget, enabling bandwidth sharing without idle resources while maintaining isolation through per-tenant transaction limits and preventing noisy neighbor effects
Expected Effect : Memory bandwidth utilization >90%; per-tenant allocation 2-4GB maintained; context switching overhead <5%
Risk Control :
- Transaction limit calibration accuracy across heterogeneous workloads
- latency variance during credit reallocation transitions
- picker logic fairness under sustained oversubscription
Problem Direction 4 :
ImproveMemory allocation per tenant
VSConstraintMust not deteriorate
Inspiration 1 : Cross-domain reference
Application Principle: #15 Dynamics
Cross-domain applicability
This patent improves compression efficiency (quantity of stored data) while maintaining image quality by dynamically adjusting bit allocation per channel. It demonstrates how [dynamic resource allocation] resolves the contradiction between maximizing resource utilization and maintaining quality constraints, directly paralleling the memory allocation challenge of balancing tenant capacity against context switching overhead.
Image data compression
Innovative Solution Refine solution
Elastic Memory Allocation with Tenant Activity-Based Expansion-Compression Cycles
Allocate memory dynamically per tenant activity phase
How to solve :
- Implement three-state memory lifecycle: allocate 8-10GB during active token generation (50-150ms bursts), compress to 2GB during idle periods (>500ms), and 0.5GB for dormant tenants
- state transitions triggered by request queue depth and last-access timestamp
- Deploy fast compression engine using LZ4 algorithm achieving 3:1 ratio at 2GB/s throughput, compressing KV cache within 15ms during idle transition
- decompression pipeline pre-fetches next active tenant during current tenant's compute phase
- Maintain rotating buffer pool of six 8GB slots for 24 total tenants: 6 active (8GB each), 12 idle (2GB each), 6 dormant (0.5GB each)
- average allocation 3.5GB per tenant, supporting 24 tenants on 84GB total memory versus 10 tenants with static 8GB allocation
Expected Effect : Context switch overhead <5%; tenant capacity +140%; bandwidth utilization 89%
Risk Control :
- compression latency spike during burst transitions
- decompression pipeline stall on cache miss
- memory fragmentation from repeated allocation cycles
Inspiration 2 : Technology in this field
Search: Dynamic memory overallocation, Context switch overhead reduction, Multi-tenant resource allocation, Shared memory management, Dynamic memory bandwidth control
Existing SolutionRefine solution
Dynamic Memory-Mapping Overallocation with TLB-Aware Placement for Multi-Tenant LLM Context Management
Allocate memory using TLB-optimized array placement where tenant memory structures are organized to minimize TLB eviction events during context switches, achieving large-allocation benefits within small physical footprints
How to solve :
- Implement dynamic memory-mapping overallocation by creating arrays of related MMU mappings placed together, skipping array elements that would cause TLB overcommitment, and scanning for array-start entries that avoid TLB contention (reference index 1 methodology)
- Establish upper bounds for dynamic structures per tenant based on workload profiling, then place structures within intentionally oversized arrays with reserved unused address ranges to maximize TLB utilization without physical memory waste
- Deploy per-tenant memory windows with read-only shared executable mappings and dedicated writable regions, enabling simultaneous access to all tenant memory without TLB replacements or context flush operations, reducing switching overhead from 87 cycles to 5 cycles as demonstrated in hardware security module implementations
Expected Effect : Context switching overhead reduced to <5% of cycles; effective memory utilization equivalent to 8-12GB allocation while maintaining 2-4GB physical footprint per tenant; tenant capacity maintained at small-allocation levels
Risk Control :
- TLB slice distribution complexity across processor architectures
- memory fragmentation from intentionally skipped array elements
- runtime overhead of maintaining allocation record registries
