Optimize LLM KV Cache Compression for Memory-Bound Inference

Overview of Technical Issues:

The KV cache storage module excessively consumes GPU memory as sequence length increases, creating a memory-bound bottleneck that limits inference batch size and throughput; meanwhile, the compression processing unit insufficiently reduces cache footprint while preserving model quality, failing to alleviate the memory constraint; the goal is to achieve effective compression that significantly reduces memory usage while maintaining inference accuracy and speed.

Solution directions generated for this problem

Problem Direction 1 :

ImproveCompression ratio effectiveness
VS
ConstraintCompression computational overhead

Inspiration 1 : Cross-domain reference

Application Principle: #35 Parameter changes
Cross-domain applicability Assess applicability
This patent improves quantity of substance (water production capacity) by incorporating high-efficiency heat exchangers that reduce required compression ratios below 1.11, thereby preventing deterioration of energy use by moving object (compressor energy consumption). It demonstrates how [parameter changes] in system operating conditions—specifically lowering compression requirements through enhanced heat transfer—enables higher throughput with reduced energy input, directly paralleling the current need to increase compression ratio effectiveness without increasing computational overhead.
Mechanical vapour compression arrangement having a low compression ratio
Innovative Solution Refine solution

Workload-adaptive dual-mode KV cache compression with phase-triggered switching

Adaptive dual-mode compression switches between phases based on inference workload
How to solve :
  • Implement phase-detection logic that monitors token generation rate: prefill phase (batch processing) triggers 8× INT4 quantization mode, decode phase (per-token generation) triggers 2× INT8 mode, switching threshold set at token rate <10 tokens/sec for decode detection
  • Deploy hardware-accelerated quantization kernels using Tensor Core INT4 matrix operations during prefill (compression overhead 8-12% of phase time, amortized over batch), standard INT8 ops during decode (overhead <3% per token)
  • Establish compression state transition protocol: monitor GPU memory utilization every 50ms, if exceeding 75% during decode, temporarily revert 10% oldest tokens to 4× compression, maintaining dynamic balance between memory pressure and latency
Expected Effect : 8× compression in prefill, 2× in decode, net overhead <5%, memory footprint reduced to 35-42%, batch size increased 3.5-4×, attention score deviation <1.8%
Risk Control :
  • phase detection latency causing mode-switch delay
  • quantization kernel compatibility across GPU architectures
  • memory thrashing during frequent state transitions

Inspiration 2 : Technology in this field

Search: GPU-based parallel compression, adaptive entropy coding, FPGA hardware acceleration, ratio-quality modeling, prediction-based compression
Existing SolutionRefine solution

Adaptive Multi-Level Escape Token Entropy Coding for KV Cache Compression

Apply adaptive entropy coding with multi-level escape tokens to KV cache data, enabling efficient compression without increasing computational burden
How to solve :
  • Implement multi-level escape tokens (Escape-1, Escape-2, ..., Escape-n) in entropy coding to coalesce multiple context escapes into single tokens, reducing bit overhead
  • Separate compressed output into encoded symbols stream and compressed literals block positioned at stream beginning/end to eliminate fractional bit loss at byte boundaries and enable parallel decompression
  • Utilize adaptive probability models that update context weights during encoding/decoding to maintain compression efficiency across varying sequence lengths without pre-generating full codebooks
Expected Effect : Compression ratio improvement 15-25% with negligible encoding overhead; decompression speed increase via reduced escape token processing
Risk Control :
  • Context model adaptation convergence speed
  • memory overhead for maintaining adaptive probability tables
  • compatibility with existing KV cache architectures

Problem Direction 2 :

ImproveCompression ratio effectiveness
VS
ConstraintDecompression latency

Inspiration 1 : Cross-domain reference

Application Principle: #10 Preliminary action
Cross-domain applicability Assess applicability
This patent improves data throughput (Quantity of substance) by using [preliminary action] to cache tree data before decompression, preventing Loss of time deterioration. It directly addresses the contradiction by performing parsing work in advance and reusing results, hiding latency off the critical path—matching the current need to increase compression ratio without adding decompression delay.
Technologies for performing low-latency decompression with tree caching
Innovative Solution Refine solution

Pipelined KV cache pre-decompression with asynchronous staging

Overlap decompression with computation
How to solve :
  • Implement asynchronous decompression pipeline: while layer N computes attention, GPU async streams decompress layer N+1's KV cache in parallel, hiding decompression latency off critical path
  • Deploy dual-buffer staging architecture: maintain two decompression buffers (each 512 tokens capacity) alternating between decompression and consumption states, ensuring attention always reads from pre-decompressed buffer
  • Apply 8× INT4 quantization for stored cache with hardware-accelerated dequantization kernels (Tensor Core INT4→FP16 conversion at 400 GB/s bandwidth), completing 2048-token decompression in 1.2ms while attention computation takes 8-12ms per layer
Expected Effect : Decompression latency <2% per layer; 7.5× compression ratio; batch size +4×; throughput +3.2×
Risk Control :
  • pipeline synchronization failure under variable sequence length
  • buffer overflow during attention spikes
  • async stream scheduling overhead on older GPUs

Inspiration 2 : Technology in this field

Search: Variable-length coding schemes, Parallel decompression architecture, Chunk-based compression, Adaptive compression ratio selection, Arithmetic coding optimization
Existing SolutionRefine solution

Non-Zero Bit Selection (NBS) Coding for KV Cache Compression

Apply variable-length coding with parallel decompression architecture to KV cache compression
How to solve :
  • Implement Non-Zero Bit Selection (NBS) scheme that enables highly parallel decompression achieving 3-cycle decompression for 8×8 blocks versus 12 cycles for exponential Golomb code, reducing decompression time to 25% while maintaining equivalent compression ratios
  • Design parallel decompression hardware units with dedicated bit-selection logic that processes multiple cache blocks simultaneously, separating coding and modeling stages to optimize decompression throughput without deteriorating compression efficiency
  • Integrate adaptive block-level compression where KV cache tensors are partitioned into fixed-size blocks (8×8 or 16×16), each compressed independently to enable random access and minimize decompression overhead during inference, with compression ratio monitored per block to trigger exception handling when ratio falls below threshold (e.g., <2:1)
Expected Effect : Decompression latency reduced to 25% of baseline; compression ratio maintained at 20-30:1 for typical KV cache patterns; inference throughput increased by 2.7-4× through reduced memory bandwidth
Risk Control :
  • Hardware implementation complexity for parallel decompression units
  • Block size selection trade-off between compression efficiency and access granularity
  • Exception handling overhead for low-compressibility blocks

Problem Direction 3 :

ImproveCompression ratio effectiveness
VS
ConstraintInformation preservation accuracy

Inspiration 1 : Cross-domain reference

Application Principle: #3 Local quality
Cross-domain applicability Assess applicability
This patent improves compression efficiency (quantity of encoded substance) by partitioning video blocks into multiple transform units with differentiated encoding modes, while preventing loss of information through adaptive entropy encoding based on coefficient properties. It demonstrates [local quality] by applying heterogeneous compression strategies tailored to regional content characteristics, directly matching the current contradiction.
Method for partitioning block and decoding device
Innovative Solution Refine solution

Layer-adaptive heterogeneous KV cache compression with attention-weighted quantization

Stratify cache by attention impact
How to solve :
  • Partition KV cache into three compression zones by layer criticality: apply INT4 (8× compression) to middle layers 40-60%, INT8 (2× compression) to first/last 20% layers, FP16 (1× compression) to top-5% high-attention tokens identified via attention score profiling during prefill phase
  • Implement per-layer attention deviation monitoring: measure cosine similarity between compressed and baseline attention scores every 128 tokens, dynamically upgrade compression from INT4→INT8 if deviation exceeds 1.5%, downgrade from INT8→INT4 if below 0.8%
  • Deploy value-key asymmetric quantization: compress value vectors to INT4 (less attention-sensitive), retain key vectors at INT8 (critical for score calculation), achieving 5.2× average compression while maintaining attention fidelity within 1.8% deviation across all layers
Expected Effect : 5.2× compression ratio, <1.8% attention deviation, memory footprint reduced to 38%
Risk Control :
  • attention profiling overhead in prefill phase
  • dynamic quantization switching latency
  • per-layer deviation tracking accuracy

Inspiration 2 : Technology in this field

Search: Low-bit Quantization (INT4/FP4), KV-Cache Compression, Adaptive Precision Control, Post-Training Quantization, Outlier-Aware Quantization
Existing SolutionRefine solution

Tiered KV Cache with Runtime-Certified Adaptive Precision Fallback

Deploy tiered storage architecture with INT8 keys and INT4 values in GPU VRAM while retaining FP16 originals in system RAM as deterministic fallback layer
How to solve :
  • Implement two-term error decomposition computing per-head per-step bounds on (i) attention distribution distortion from key quantization and (ii) value reconstruction error, with online bound computation driving adaptive precision selection
  • establish multi-stage fallback ladder (INT4→INT8→FP16) triggered when computed error bounds exceed task-specific tolerances (e.g., ≤0.01 for retrieval, ≤0.05 for generation)
  • integrate runtime certification logic ensuring each attention operation either satisfies bounded-error guarantee relative to FP16 reference or executes exact FP16 fallback recovery
Expected Effect : 75% memory reduction (INT4+INT8) with zero quality degradation on NIAH/RULER benchmarks; catastrophic failure elimination in value-sensitive tasks
Risk Control :
  • Per-head error bound computation overhead
  • system RAM bandwidth saturation during fallback
  • tolerance threshold calibration across task diversity

Problem Direction 4 :

ImproveCompression ratio effectiveness
VS
ConstraintMust not deteriorate

Inspiration 1 : Cross-domain reference

Application Principle: #15 Dynamics
Cross-domain applicability Assess applicability
This patent improves compression efficiency (quantity of compressed data) by dynamically selecting transformation unit sizes according to image characteristics, preventing deterioration of encoding overhead and processing speed. It demonstrates the Dynamics principle by adapting compression granularity to content needs, directly paralleling the current need to dynamically adjust compression ratio between memory-critical and latency-critical phases.
Method and apparatus for encoding and decoding image by using large transformation unit
Innovative Solution Refine solution

Inference-stage adaptive KV cache compression with dynamic ratio switching

Switch compression ratio dynamically across inference stages to separate memory and latency demands
How to solve :
  • Apply 8× INT4 quantization during prefill phase when memory-bound — compress all KV pairs with per-channel scaling factors stored separately, tolerate 12-18ms decompression overhead per batch
  • Switch to 2× INT8 quantization during decode phase when latency-critical — use lookup-table dequantization requiring only 0.8-1.2ms per token, maintaining real-time responsiveness
  • Implement stage detector module monitoring sequence position and batch size — trigger ratio transition at prefill-to-decode boundary (typically at position 512-1024 tokens), with transition completing within 3ms via parallel re-quantization of recent 256 tokens
Expected Effect : Memory footprint reduced to 35-42% at 2048 tokens; decode latency overhead <6%; batch size increased 3.8-4.2×; attention score deviation <1.8%
Risk Control :
  • transition timing mismatch causing temporary memory spike
  • quantization scale factor precision loss during re-quantization
  • hardware INT4 Tensor Core availability varies across GPU generations

Inspiration 2 : Technology in this field

Search: Cache compression algorithms, Compression ratio metrics, Compressibility-based cache management, Compression latency tradeoffs, Memory compression systems
Existing SolutionRefine solution

Adaptive Multi-Tier Compressed KV Cache with Dynamic Algorithm Selection

Implement hierarchical KV cache architecture with compressed lower tiers and uncompressed higher tier, dynamically selecting compression algorithms based on cache fullness and access patterns
How to solve :
  • Establish multi-tier cache hierarchy with uncompressed L1 tier (fast access, 10-20% capacity) and 2-3 compressed lower tiers using algorithms of varying ratios: tier-2 with moderate compression (BDI/FPC, 2:1 ratio, 3-5 cycle latency), tier-3 with aggressive compression (LZH-based, 4:1 ratio, 10-15 cycle latency)
  • Implement compression selection component that monitors cache fullness levels and selects algorithms dynamically—use lightweight compression (≤2:1 ratio) when cache <50% full, moderate compression (2-3:1 ratio) at 50-80% fullness, aggressive compression (≥3:1 ratio) when >80% full to prioritize eviction speed
  • Deploy hit-score-based eviction policy considering compression factors, where entries with higher compression ratios receive elevated retention priority, and perform parallel decompression using multiple decoders for bandwidth optimization
  • Integrate quality control checkpoints: validate decompressed data integrity via CRC-32 checksums (error rate <10⁻⁶), monitor decompression latency per tier (L2 ≤5 cycles, L3 ≤15 cycles), measure effective compression ratio per algorithm (target: 2.5-3.5:1 average), and track inference accuracy degradation (tolerance: <0.5% BLEU/perplexity loss)
Expected Effect : Memory footprint reduction 60-70%; decompression latency <10 cycles average; inference accuracy maintained within 0.3% degradation
Risk Control :
  • Algorithm switching overhead during dynamic selection
  • decompression latency variance across workload patterns
  • metadata storage overhead for multi-tier management
Patsnap Eureka Solution