Token Skipping Plugin for Structured LLM Output Accuracy

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

State-of-the-art transformer-based language models consume significant computational resources and lack accuracy due to their probabilistic nature, limiting their application in situations requiring strictly formatted output.

Innovation Solution

A computing system with a model plugin that identifies deterministic input tokens and skips their transmission to the machine learning model, writing them directly to the output sequence while passing the remainder to the model for probabilistic token-wise generation, thereby interleaving deterministic and probabilistic tokens in the output.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Manufacturing precision

If transformer-based language models are used for text generation, then output accuracy and format adherence improve, but computational resource consumption increases significantly

Engineering Contradiction:

Improveoutput accuracyVSAvoidcomputational resource consumption

Core Design Contradiction:

Manufacturing precisionVSUse of energy by moving object

Solution Approach 1:

The patent segments the token generation process into two distinct parts: deterministic token generation (for format-adherent tokens) and probabilistic token generation (for creative/content tokens). This segmentation allows the system to apply different processing strategies to different portions of the output, reducing overall computational load while maintaining accuracy where needed.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies local quality by treating different positions and types of tokens differently in the sequence. Format-adherent tokens receive deterministic processing ensuring structural accuracy, while content tokens receive probabilistic processing for creativity. This localized approach optimizes resource usage by applying heavy computation only where necessary.

Inventive Principle:

Principle #3Local quality

2Adaptability or versatility

If probabilistic token-wise generation is used, then creative output is achieved, but output stability and format adherence deteriorate

Engineering Contradiction:

Improvecreative outputVSAvoidoutput stability

Core Design Contradiction:

Adaptability or versatilityVSStability of the object's composition

Solution Approach 1:

The patent implements dynamics by making the generation process adaptive and flexible. The system dynamically switches between deterministic and probabilistic modes based on the token type and position, allowing the output to be both stable (in format) and creative (in content). This dynamic approach resolves the contradiction between stability and adaptability.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The patent introduces an intermediary mechanism (the model plugin) that sits between the deterministic structure requirements and the probabilistic generation process. This intermediary identifies format-adherent tokens and applies appropriate generation strategies, mediating between the conflicting requirements of stability and creativity.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Productivity

If all input tokens are processed by the machine learning model, then complete output generation is achieved, but processing time and computational cost increase

Engineering Contradiction:

Improveoutput generation completenessVSAvoidprocessing time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The patent applies preliminary action by pre-identifying format-adherent tokens in the output sequence before actual generation. The model plugin analyzes the structured output definition in advance, marks tokens that require deterministic generation, and prepares the generation strategy beforehand. This preliminary classification enables more efficient processing during actual token generation.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent implements skipping by bypassing the probabilistic generation process for format-adherent tokens. Once these tokens are identified, the system rushes through their generation using deterministic methods, skipping the time-consuming probabilistic sampling and selection steps. This selective skipping significantly reduces processing time for structured portions of the output.

Inventive Principle:

Principle #21Skipping (Rushing through)

Data Source

PatentUS20250356124A1Machine learning model with input token skipping and insertion

Publication Date: 2025.11.20 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

A computing system is provided that instantiates a trained machine learning model and a model plugin. During inference, the model plugin receives an input sequence of input tokens of a prompt including context and a structured output definition. When the model plugin identifies deterministic input tokens corresponding to the structured output definition, it skips transmission of the deterministic input tokens to the machine learning model, and writes the one or more deterministic input tokens as deterministic output tokens to an output token sequence. The model plugin further passes a remainder of input tokens in the input sequence to the machine learning model. The machine learning model performs probabilistic token-wise generation of other output tokens in the output sequence based on the remainder of the input tokens, and outputs the output sequence including the deterministic output tokens and the other output tokens generated by the probabilistic token-wise generation.