How to Select LLM Context Compression Method for Memory-Constrained Edge Devices
Overview of Technical Issues:
Under strict memory constraints on edge devices, existing context compression methods insufficiently balance compression ratio against semantic information preservation—aggressive compression causes the LLM inference engine to receive inadequate context and produce degraded outputs, while insufficient compression exceeds available memory capacity; the goal is to select a compression method that enables reliable LLM deployment within memory limits while maintaining acceptable inference quality.
Solution directions generated for this problem
Problem Direction 1 :
ImproveContext compression ratio
VSConstraintInference output reliability
Inspiration 1 : Cross-domain reference
Application Principle: #1 Segmentation
Cross-domain applicability
This patent improves bandwidth efficiency (analogous to memory footprint/Quantity of substance) by [segmenting] video frames into blocks with selective compression intensity, while preventing image quality degradation (analogous to Reliability). It demonstrates how [Segmentation] enables differentiated treatment—critical blocks preserve fidelity while non-critical blocks compress aggressively—directly mirroring the current contradiction of maximizing compression ratio without sacrificing inference reliability.
Method and apparatus for processing video
Innovative Solution Refine solution
Hierarchical semantic block compression with criticality-based retention for edge LLM deployment
Divide context into semantic blocks with criticality scoring
How to solve :
- Parse input context into semantic blocks (task instructions, entity references, background, examples) using dependency parsing
- assign criticality scores 0-1 via attention weight analysis from a lightweight scoring model (50MB overhead)
- Apply differentiated compression: retain task instructions and high-score entities at 30% compression (BPE tokenization), compress background/examples at 85% compression (abstractive summarization + 8-bit quantized embeddings), achieving 72% overall compression within 600MB memory budget
- Implement runtime quality monitoring: track output confidence scores per inference cycle
- if confidence drops below 0.85 threshold, dynamically decompress next-priority semantic blocks from storage (100MB reserve buffer), adjusting compression to 65% until reliability recovers above 85%
Expected Effect : Compression ratio 72%, memory 600MB, reliability ≥85%, 40% improvement over uniform compression
Risk Control :
- criticality scoring model accuracy <0.75 causes misclassification
- summarization quality variance across domains
- decompression latency exceeds 200ms impacts real-time inference
Inspiration 2 : Technology in this field
Search: Context compression techniques, Hallucination detection and mitigation, Edge device optimization, Reliability verification methods, Dynamic compression ratio
Existing SolutionRefine solution
Instruction-Aware Contextual Compression with Self-Information Guided Token Pruning
Compress context by selectively retaining high self-information tokens guided by instruction awareness to preserve semantic completeness within memory limits
How to solve :
- Implement self-information calculation for each token using -log(P(token|context)) from LLM probability distributions, where tokens with self-information above adaptive threshold (tuned per instruction type) are preserved
- Apply instruction-aware compression by parsing user query to identify critical entities/concepts, then segment context into manageable chunks (256-512 tokens), generate embeddings, and prioritize chunks with cosine similarity >0.7 to instruction semantics
- Deploy dynamic compression ratio allocation where informative segments retain 40-50% tokens while redundant segments compress to 10-20%, achieving overall 70-80% compression while maintaining >85% task-critical information as validated by Rouge-1 scores with <0.05 degradation
Expected Effect : Context memory reduced 50-75%; inference speed increased 2.2x; output reliability maintained >85%
Risk Control :
- Self-information threshold calibration across diverse instruction types
- computational overhead of per-token probability extraction on resource-constrained devices
- semantic boundary detection accuracy in multi-topic contexts
Problem Direction 2 :
ImproveSemantic information preservation rate
VSConstraintMemory consumption
Inspiration 1 : Cross-domain reference
Application Principle: #2 Taking out (Extraction)
Cross-domain applicability
This patent improves loss of information (retaining critical physiologic data) while preventing deterioration of volume of stationary object (limited memory capacity) by [extracting] and storing only priority-indexed essential data, directly paralleling the current contradiction of preserving semantic completeness within constrained edge device memory.
Apparatus for data retention in an implantable medical device
Innovative Solution Refine solution
Semantic-Priority Extraction with Tiered Context Storage for Edge LLM Deployment
Extract semantic core for memory-efficient LLM inference
How to solve :
- Deploy semantic extraction module that identifies and extracts only task-critical elements (entities, relations, instructions) from full context, discarding syntactic scaffolding and redundant phrasing to achieve 72% memory reduction
- Implement three-tier storage architecture: Tier-1 stores extracted semantic core (200-300MB, 95% preservation), Tier-2 stores compressed summaries of non-critical context (150-200MB, 60% preservation), Tier-3 discards redundant data—total memory 350-500MB
- Use importance scoring algorithm (TF-IDF + named entity density + task-keyword matching) with threshold ≥0.65 to classify context segments, ensuring >85% semantic completeness for inference while fitting within 512MB limit
Expected Effect : Memory usage 350-500MB, semantic preservation >85%, inference accuracy maintained at 83-87%
Risk Control :
- extraction algorithm misclassifies critical context segments
- importance scoring threshold requires domain-specific calibration
- tier-1 storage overflow under complex multi-turn dialogues
Inspiration 2 : Technology in this field
Search: Semantic information preservation, Memory-efficient semantic storage, Attention-based compression, Semantic completion methods, Knowledge graph compression
Existing SolutionRefine solution
Attention-Guided Semantic Token Selection with Syntactic Boundary Preservation for Edge LLM Context Compression
Deploy attention-driven semantic compression that preserves task-critical information within memory limits while maintaining inference quality
How to solve :
- Compute token-level attention scores s*c across all transformer heads/layers to identify semantic salience, select top-p fraction (p=0.15-0.25) tokens forming high-attention set Ĉ
- Apply SemanticPreserve operation expanding selected tokens to syntactic boundaries (commas, periods, conjunctions) using tokenizer offset maps, merging overlapping spans into coherent sub-sentential units to prevent fragmented phrases
- Encode preserved spans via single-layer graph neural network capturing semantic relationships between context segments, store compressed graph representation (node embeddings + adjacency matrix) requiring 60-75% less memory than full token sequences
Expected Effect : Compression ratio 72-78%, semantic preservation >87%, memory footprint 580MB-1.8GB
Risk Control :
- Attention score calibration across model architectures
- syntactic boundary detection accuracy for non-English languages
- graph encoding overhead on resource-constrained edge processors
Problem Direction 3 :
ImproveContext compression ratio
VSConstraintMust not deteriorate
Inspiration 1 : Cross-domain reference
Application Principle: #3 Local quality
Cross-domain applicability
This patent improves quantity of substance (reduces feedback overhead from 5 bits to fewer bits) while preventing deterioration of information quality (maintains effective channel assessment) by applying local quality principle through sub-tables with different resolutions. It demonstrates selective compression intensity based on content criticality, directly paralleling the need for high overall compression ratio while controlling semantic loss through locally-adapted compression strategies.
Methods and apparatus for channel quality indication feedback in a communication system
Innovative Solution Refine solution
Semantic-priority tiered context compression with adaptive resolution tables
Tiered compression with adaptive resolution
How to solve :
- Classify context into three semantic tiers: Tier-1 (task-critical entities, instructions) at 35% compression, Tier-2 (supporting facts) at 65% compression, Tier-3 (background, examples) at 85% compression, achieving 72% overall compression while preserving >85% task-relevant semantics
- Implement resolution-adaptive compression tables per tier: Tier-1 uses 16-bit float embeddings with 512-entry codebook, Tier-2 uses 8-bit quantized vectors with 256-entry codebook, Tier-3 uses 4-bit sparse indices with 64-entry codebook, each optimized for local semantic density
- Deploy real-time tier promotion mechanism: monitor LLM attention scores during inference
- if Tier-2/3 tokens receive attention weight >0.15, dynamically decompress to next-higher resolution within 50ms latency budget, maintaining output confidence ≥85%
Expected Effect : Memory 580MB (72% compression), semantic preservation 87%, inference accuracy 86%
Risk Control :
- codebook training requires 10K+ domain samples
- tier classification accuracy <92% degrades performance
- decompression latency spikes under high promotion frequency
Inspiration 2 : Technology in this field
Search: Context compression with learned tokens, KV cache compression, Semantic-aware lossy compression, Quantization-based model compression, Compression-aware memory architecture
Existing SolutionRefine solution
Semantic-Aware Hierarchical Context Pruning with Application-Context Registration
Application context identifies semantically critical data attributes via XPath registration to preserve essential tokens during compression
How to solve :
- Implement application-context registration module that extracts XPath expressions from client application requirements, identifying which context elements (e.g., employee.name.first, contact.email) are semantically essential for inference tasks
- Build pruning tree structure using buildTree algorithm on registered paths, applying recursive head/tail path decomposition to construct hierarchical compression map matching context token tree to application data demands
- Execute selective token preservation by traversing context XML/hierarchical representation, retaining only black nodes matching pruning tree entries while removing unused branches, achieving compression ratios of 70-80% (e.g., 4KB to 1KB blocks) by eliminating semantically irrelevant tokens
Expected Effect : Memory footprint reduction 75%, semantic preservation >85%, inference quality maintained
Risk Control :
- Application context extraction accuracy and completeness
- Pruning tree construction overhead on resource-constrained devices
- Compatibility across diverse LLM context formats beyond XML hierarchical structures
Problem Direction 4 :
ImproveSemantic information preservation rate
VSConstraintMust not deteriorate
Inspiration 1 : Cross-domain reference
Application Principle: #10 Preliminary action
Cross-domain applicability
This patent improves loss of information (ensuring all devices receive necessary access updates) while managing system resource constraints through [preliminary action]—pre-defining metadata classes and queuing events offline before runtime delivery. This separates preparation from execution, allowing high-fidelity information transfer despite distributed system limitations, directly echoing the storage-inference contradiction.
Managing change events for devices in an enterprise system
Innovative Solution Refine solution
Offline semantic graph pre-compilation with on-demand runtime reconstruction for edge LLM inference
Pre-compile context into semantic graphs offline then reconstruct on-demand during inference
How to solve :
- Offline pre-compilation phase: Parse full context into knowledge graph format (entity-relation-entity triples), store as compressed adjacency matrix (INT8 quantization) plus entity embedding dictionary (FP16), achieving 72% compression to 512MB
- Deploy lightweight reconstruction engine (35MB footprint) that accepts query tokens and traverses graph to extract relevant subgraph, decompressing only task-critical nodes and edges to full semantic representation
- Implement query-driven selective decompression: score graph nodes by cosine similarity to query embedding (threshold ≥0.75), reconstruct high-scoring subgraph to >85% semantic completeness within 120ms latency, discard after inference cycle completes
Expected Effect : Storage 512MB, inference completeness 87%, latency +18% vs. uncompressed baseline
Risk Control :
- graph parsing accuracy for complex syntax
- reconstruction latency exceeding real-time budget
- entity embedding drift during quantization
Inspiration 2 : Technology in this field
Search: Model Quantization and Compression, Low-Rank Adaptation (LoRA), KV Cache Optimization, Activation-Aware Weight Quantization, Knowledge Distillation
Existing SolutionRefine solution
Outlier-Aware Mixed-Precision Context Memory for Edge LLM Inference
Store most context compactly but preserve semantically sensitive parts in higher precision for reconstruction during inference.
How to solve :
- Profile context encoder and KV-cache related operators with outlier-ratio statistics
- mark slices whose activation outlier proportion exceeds a preset threshold as high-precision, keep others in low-bit form, using W8A8 INT8 SQ or W4A16 AWQ baseline mapping
- Apply mixed-precision storage so sensitive context spans use FP16/BF16 activations with INT8 weights, ordinary spans use INT8 or INT4 storage
- perform subgraph fusion after precision remapping to reduce conversion overhead and fit 512MB-class edge memory
- Establish closed-loop quality control by calibrating on a validation set, iteratively lowering or raising the outlier threshold until semantic completeness exceeds acceptance target
- inspect outlier ratio, perplexity drift, answer consistency, and memory occupancy before deployment
Expected Effect : memory within 512MB class;semantic completeness above 85%;accuracy loss within 0.5%;performance rollback below 5%
Risk Control :
- threshold miscalibration
- operator precision mapping compatibility
- runtime precision-conversion overhead
