Self-Speculative Decoding for Faster Edge LLM Token Generation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Large language models (LLMs) face challenges in efficient deployment on resource-constrained edge platforms due to high inference times and significant latencies, even with compact models, limiting their use in conversational AI agents.

Innovation Solution

A method involving speculative decoding and model compression techniques, such as pruning and replacing transformer blocks with low-parameter replacements, allows for faster token generation without additional parameters, using a small draft model for initial predictions and a large model for verification, and applying weight-sharing and low-rank adapters to maintain accuracy.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If traditional autoregressive decoding is used, then model accuracy is maintained, but inference time increases and throughput decreases

Engineering Contradiction:

Improveinference throughputVSAvoidinference time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The draft model performs preliminary token predictions in advance, generating candidate tokens that are then verified by the main model in parallel, rather than waiting for sequential autoregressive generation

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

A smaller draft model is created as a simplified copy of the main model, using weight sharing and low-rank adapters to replicate essential prediction capabilities at reduced computational cost

Inventive Principle:

Principle #26Copying

2Loss of time

If model size is reduced for edge deployment, then latency decreases, but model accuracy and performance deteriorate

Engineering Contradiction:

ImprovelatencyVSAvoidmodel accuracy

Core Design Contradiction:

Loss of timeVSReliability

Solution Approach 1:

The system is divided into two segments: a small draft model for rapid token generation and a full-size main model for verification, allowing the draft model to operate efficiently on edge devices while the main model ensures accuracy

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The draft model uses parameter compression techniques including weight sharing across model layers and low-rank adapter transformations, changing the parameter representation to reduce memory footprint while preserving functional capabilities

Inventive Principle:

Principle #35Parameter changes

3Reliability

If full model parameters are used, then prediction accuracy is maintained, but device memory requirements and computational complexity increase

Engineering Contradiction:

Improveprediction accuracyVSAvoidmodel parameter count

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

Model weights serve multiple functions: the draft model uses compressed weights for rapid inference, while the main model uses full weights for verification, allowing the same parameter set to fulfill both speed and accuracy requirements

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

Transformation matrices use low-rank decomposition to represent complex weight relationships with fewer parameters, changing the parameter space from full-rank to low-rank representation while maintaining predictive power

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS20250384273A1Efficient self-speculative decoding architecture for increasing LLM inference throughput

Publication Date: 2025.12.18 SAMSUNG ELECTRONICS CO LTD

AI summary

A method of generating a token for a language model includes obtaining a language model comprising one or more transformer blocks, training the language model based on one or more parameters, identifying a first parameter, from among the one or more parameters, to compress or remove from the language model, finetuning the language model based on the first parameter being compressed or removed, and providing the finetuned language model to an electronic device.