Optimize LLM Batch Size for GPU Memory Utilization

Overview of Technical Issues:

The GPU memory allocation unit insufficiently adapts its storage capacity to balance batch size against available VRAM, causing either memory overflow and system crashes when batches exceed capacity, or underutilized compute resources and reduced throughput when batches are too conservative; the goal is to dynamically optimize batch sizing to maximize GPU memory utilization while maintaining stable inference operations.

Solution directions generated for this problem

Problem Direction 1 :

ImproveMemory allocation adaptability
VS
ConstraintAllocation system complexity

Inspiration 1 : Cross-domain reference

Application Principle: #1 Segmentation
Cross-domain applicability Assess applicability
This patent applies [Segmentation] by dividing memory management into independent architectural modes with selective feature enablement, improving adaptability across diverse memory scenarios while preventing complexity escalation through modular facility control that isolates DAT logic from base operations, directly echoing the contradiction of enhancing VRAM allocation versatility without expanding system complexity.
Managing processing associated with selected architectural facilities
Innovative Solution Refine solution

Zone-partitioned memory allocator with independent module governance for 10MB-40GB VRAM range

Partition VRAM range into independent zones with dedicated allocators
How to solve :
  • Divide 10MB-40GB range into four discrete zones: Zone-A (10MB-1GB), Zone-B (1-10GB), Zone-C (10-25GB), Zone-D (25-40GB), each governed by an independent allocator module with zone-specific logic
  • Implement zone-specific batch size tables: Zone-A uses fixed sizes [1,2,4], Zone-B [4,8,16], Zone-C [16,32,64], Zone-D [64,128,256], eliminating complex continuous calculation—each module <500 lines of code
  • Deploy lightweight zone selector at runtime: query GPU driver's available VRAM via single API call (cudaMemGetInfo, <0.1ms overhead), index pre-computed zone table, activate corresponding allocator—total decision latency <1ms, no predictive algorithms required
Expected Effect : Codebase +40% vs 3-5× baseline; allocation decision <1ms; zone switching overhead <2% GPU cycles; ±50MB precision via zone-table lookup
Risk Control :
  • zone boundary transition instability during workload shift
  • pre-computed table accuracy degradation with model architecture changes
  • zone selector failure causing fallback to conservative default

Inspiration 2 : Technology in this field

Search: Adaptive memory pool tuning, Workload-based allocation prediction, Multi-scale block management, Runtime memory resizing, Fragmentation-aware allocation
Existing SolutionRefine solution

Problem Direction 2 :

ImproveVRAM utilization monitoring precision
VS
ConstraintMeasurement overhead impact on throughput

Inspiration 1 : Cross-domain reference

Application Principle: #24 Intermediary
Cross-domain applicability Assess applicability
This patent improves scheduling accuracy by using [predicted channel information as an intermediary] instead of real-time measurement reports, thereby improving precision while preventing measurement overhead and time loss from worsening. It directly echoes the current contradiction of achieving measurement precision while avoiding loss of time through GPU cycle overhead.
User equipment scheduling methods and equipment
Innovative Solution Refine solution

Hardware-assisted memory event interception for zero-overhead layer-level VRAM tracking

Use GPU memory as intermediary proxy for tracking
How to solve :
  • Leverage GPU Memory Management Unit (MMU) page fault counters as passive observation points — configure 4KB page granularity to track layer allocation events without active profiling code injection
  • Deploy driver-level memory event hooks via CUDA IPC Event Handle API to intercept cudaMalloc/cudaFree calls at layer boundaries, timestamp each event in a lock-free ring buffer (512-entry circular queue, 64-byte aligned) on host memory
  • Construct differential memory map by subtracting consecutive allocation snapshots — calculate per-layer delta with ±4KB hardware precision, aggregate to ±50MB layer-level accuracy using 10-sample moving average filter
  • Parse event stream asynchronously on dedicated CPU thread (pinned to isolated core) while GPU executes inference uninterrupted — batch size controller queries cached map via shared memory read (single atomic load, <50ns latency)
Expected Effect : Overhead <0.8% GPU cycles, latency <2ms per batch, precision ±45MB
Risk Control :
  • MMU counter overflow on rapid allocation bursts
  • ring buffer race condition under high concurrency
  • driver API compatibility across CUDA versions

Inspiration 2 : Technology in this field

Search: GPU memory resource tracking, hardware-based performance monitoring, layer-wise memory management, low-latency GPU measurement, adaptive utilization monitoring
Existing SolutionRefine solution

Hardware-Based Memory Trapping with Event Filtering for Layer-Level VRAM Monitoring

Implement hardware memory trapping function to monitor GPU memory accesses at page granularity without software polling overhead
How to solve :
  • Deploy memory trapping function within GPU memory allocator to detect and report memory page modifications (e.g., 64KB or 4KB pages) via interrupt callbacks, capturing layer-specific allocation patterns
  • Configure coarse and fine event filters to isolate memory access events by shader type (vertex/pixel/compute) and processing resource, enabling selective monitoring of inference layers while blocking irrelevant events
  • Integrate performance monitoring circuitry with counters tracking memory transactions per layer, writing results to dedicated buffer addresses accessible via memory-mapped I/O, avoiding main execution path interference
Expected Effect : ±50MB precision, <2% overhead, <5ms latency
Risk Control :
  • Memory trapping activation overhead on high-frequency allocations
  • Event filter configuration complexity across diverse layer types
  • Buffer overflow risk during burst memory operations

Problem Direction 3 :

ImproveBatch sizing responsiveness
VS
ConstraintMeasurement overhead impact on throughput

Inspiration 1 : Cross-domain reference

Application Principle: #10 Preliminary action
Cross-domain applicability Assess applicability
This patent improves Speed (rapid event detection and data transmission) while preventing Loss of time (avoiding setup delays and communication overhead) by [pre-configuring] input terminals and [pre-storing] processing instructions in memory, allowing the processor to execute transitions deterministically without real-time computation delays, directly matching the current contradiction of achieving fast batch responsiveness while minimizing recalculation overhead.
Event generation in an input module for an industrial controller
Innovative Solution Refine solution

Pre-computed batch size lookup table with GPU idle-time calibration

Pre-compute batch size lookup table during GPU idle cycles
How to solve :
  • During model loading phase, execute offline profiling to measure memory footprint for 8-12 representative batch sizes (1,2,4,8,16,32,64,128) across typical input dimensions, storing results in a hash-indexed lookup table (key: input_shape + available_VRAM, value: optimal_batch_size)
  • Runtime batch selection queries the table via single hash lookup in <0.2ms, eliminating real-time calculation
  • Trigger incremental recalibration only when free VRAM changes exceed 15% threshold, updating 2-3 table entries asynchronously on CPU thread during inference gaps without blocking GPU pipeline
Expected Effect : Batch decision latency <0.5ms, GPU overhead <0.3%, responsiveness <1s
Risk Control :
  • lookup table cache miss under novel input sizes
  • hash collision causing incorrect batch retrieval
  • asynchronous calibration thread synchronization failure

Inspiration 2 : Technology in this field

Search: Dynamic Batching, GPU Performance Monitoring, Batch Size Optimization, Inference Latency Reduction, GPU Memory Management
Existing SolutionRefine solution

Binary Search-Based Adaptive Batch Sizing with GPU Memory Profiling

Employ lightweight binary search algorithm to rapidly converge on optimal batch size by probing GPU memory boundaries between inference batches
How to solve :
  • Implement binary search batch sizing starting from safe baseline (e.g., batch=32), incrementally testing larger batches until GPU memory allocation succeeds, recording maximum feasible batch size within 3-5 iterations
  • utilize NVIDIA Management Library (NVML) or equivalent APIs to query available VRAM in real-time with <0.5ms polling overhead, triggering recalculation only when free memory changes exceed 10% threshold
  • integrate fast profiling checkpoints that measure memory footprint per sample during initial warmup (first 10-20 batches), then apply linear extrapolation formula (VRAM_needed = baseline_memory + batch_size × per_sample_memory) to predict safe batch sizes, avoiding full re-profiling
  • employ exponential backoff strategy when allocation fails, immediately reducing batch size by 25-50% to prevent system crashes, then resume binary search from new safe point.
Expected Effect : Batch size responsiveness <800ms; recalculation overhead 0.6-0.8ms; GPU cycle consumption 1.2-1.8%
Risk Control :
  • Binary search convergence stability under variable workload
  • NVML polling accuracy and driver compatibility
  • Memory fragmentation impact on prediction reliability

Problem Direction 4 :

ImproveVRAM utilization monitoring precision
VS
ConstraintAllocation system complexity

Inspiration 1 : Cross-domain reference

Application Principle: #6 Universality
Cross-domain applicability Assess applicability
This patent improves resource prediction precision (measurement accuracy) by creating a [universal] feature extraction system that serves multiple functions (worker selection, resource prediction, continuous learning) without proportionally increasing system complexity, directly echoing the current need to achieve precise VRAM monitoring while avoiding codebase expansion through [multi-functional] design.
Distributed resource-aware training of machine learning pipelines
Innovative Solution Refine solution

Multi-functional unified instrumentation layer for layer-level VRAM profiling with minimal codebase expansion

Unified instrumentation layer serves profiling, debugging, and optimization simultaneously
How to solve :
  • Deploy a single unified instrumentation hook at GPU memory allocator interface that captures allocation/deallocation events for all layers — each hook records timestamp, layer ID, memory delta, and operation type in a 32-byte struct
  • Implement multi-purpose event stream parser that generates three outputs from one data source: (1) real-time ±50MB layer-level memory map for batch sizing, (2) debugging traces for failure analysis, (3) optimization reports for model tuning — eliminates need for separate monitoring systems
  • Use ring buffer architecture (fixed 4MB size, 128K events capacity) with lock-free write and asynchronous CPU-side read — GPU overhead <0.8% cycles, latency <2ms per batch, codebase addition <500 lines versus 3000-5000 lines for separate systems
Expected Effect : ±50MB precision achieved; codebase +15% vs +300-500%; overhead <1% GPU cycles
Risk Control :
  • ring buffer overflow under burst allocation
  • parser logic synchronization failure
  • hook injection compatibility across GPU drivers

Inspiration 2 : Technology in this field

Search: Memory profiling systems, Hierarchical memory monitoring, Runtime memory tracking, Low-overhead profiling, Memory performance metrics
Existing SolutionRefine solution

Hierarchical Memory Access Monitoring with Adaptive Granularity Profiling

Deploy a hierarchical memory monitoring system using address monitoring units (AMUs) at GPU memory hierarchy levels (L1/L2 cache, VRAM), activating monitoring granularity based on risk assessment; implement lightweight managed object base classes with serialize/deserialize methods for memory state tracking, organizing monitored units into swappable groups at routine or layer granularity; utilize event-driven profiling triggers at inference batch boundaries (call/return points) to sample memory access patterns, computing size estimates from serialized object footprints rather than full memory scans, maintaining accuracy within ±50MB while limiting overhead to under 2% through selective activation of monitoring layers based on current memory pressure thresholds.
How to solve :
  • ±50MB precision with under 15% code expansion
Expected Effect : AMU hardware availability across GPU architectures; serialization overhead during peak inference loads; memory estimation accuracy degradation under fragmented allocation patterns
Risk Control :
  • 1,2,3

Problem Direction 5 :

ImproveMemory allocation adaptability
VS
ConstraintMeasurement overhead impact on throughput

Inspiration 1 : Cross-domain reference

Application Principle: #23 Feedback
Cross-domain applicability Assess applicability
This patent improves adaptability in storage resource allocation across varying memory configurations while preventing time loss through simple validity checks and response codes as feedback signals, rather than complex predictive algorithms. It directly addresses the contradiction between configuration versatility and operational latency by using lightweight feedback mechanisms to guide allocation decisions efficiently.
Configure storage class memory command
Innovative Solution Refine solution

Threshold-triggered VRAM feedback controller with pre-computed batch size table

Feedback-driven batch adjustment at thresholds
How to solve :
  • Pre-compute a batch size lookup table during model warmup (first 10 batches): profile VRAM usage at batch sizes [1,2,4,8,16,32,64], store optimal size for each 5GB VRAM availability increment (e.g., 5GB→batch 4, 10GB→batch 8, 15GB→batch 16), table generation overhead <50ms one-time cost
  • Deploy lightweight feedback monitor reading GPU driver's native VRAM telemetry every batch via cudaMemGetInfo() API (latency <0.2ms, zero profiling injection): calculate free VRAM percentage as single feedback signal, compare against two thresholds—critical low (free VRAM <15%) and safe high (free VRAM >40%)
  • Trigger threshold-based batch adjustment only when thresholds crossed: if free VRAM <15%, index lookup table for next-lower batch size
  • if free VRAM >40% for 3 consecutive batches, index next-higher size
  • otherwise maintain current batch—decision logic executes in <0.5ms via simple table lookup, no recalculation overhead, adjustment frequency self-regulates to workload stability
Expected Effect : Adaptation overhead <0.8% GPU cycles, latency <1ms per batch, VRAM utilization 75-85%, supports 10MB-40GB range
Risk Control :
  • cudaMemGetInfo() API latency spikes on certain drivers
  • lookup table insufficient for highly variable input sizes
  • threshold tuning required per model architecture

Inspiration 2 : Technology in this field

Search: GPU dynamic memory allocation, low-overhead allocation, adaptive memory management, memory fragmentation control, latency optimization
Existing SolutionRefine solution

Dynamic Memory Pool Architecture with Percentile-Based Allocation for GPU Inference

Implement pre-allocated memory pools with variable chunk sizes to eliminate runtime allocation overhead
How to solve :
  • Establish host-side pinned memory pool and device-side GPU memory pools with chunks sized in multiples of N (e.g., N=128MB for 10MB-40GB range), allocating memory blocks without triggering GPU synchronization events
  • Apply percentile-based allocation policy where batch memory grant equals the 95th percentile of actual memory consumption from prior executions, updated per batch using cached statistics to maintain <1ms recalculation overhead
  • Implement time-decay deallocation with configurable decay constant (e.g., 30-60s) to release unused chunks back to system, preventing memory pool bloat while maintaining 5-10% buffer headroom for variance
Expected Effect : <1% GPU cycle overhead, <5ms allocation latency, 90%+ VRAM utilization
Risk Control :
  • percentile threshold calibration across workload diversity
  • memory fragmentation under mixed batch sizes
  • decay constant tuning for stability-efficiency tradeoff
Patsnap Eureka Solution