Hybrid Token Prediction Model Balancing AR Accuracy and NAR Speed

Solution Overview

Problem

Existing sequence-to-sequence models for data prediction face a trade-off between autoregressive (AR) models, which provide high accuracy but are processing intensive, and non-autoregressive (NAR) models, which are efficient but less accurate, leading to latency issues in high-volume data processing.

Innovation Solution

A hybrid approach combining AR and NAR models through pre-training and fine-tuning, where initial tokens are predicted using AR for accuracy and subsequent tokens using NAR for efficiency, leveraging a self-attention mechanism to balance accuracy and efficiency.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If autoregressive (AR) models are used for data prediction, then accuracy is improved, but processing time and computational resources increase significantly

Engineering Contradiction:

Improveprediction accuracyVSAvoidprocessing time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The patent segments the sequence generation process into two distinct phases: an autoregressive phase for generating initial tokens with high accuracy, and a non-autoregressive phase for generating remaining tokens in parallel. This segmentation allows the system to apply different processing strategies to different parts of the sequence, optimizing both accuracy and efficiency.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system dynamically switches between autoregressive and non-autoregressive processing modes based on the generation stage. The transition from AR to NAR mode is controlled by a predefined threshold or condition, allowing the model to adapt its processing strategy during the generation process to balance accuracy and speed requirements.

Inventive Principle:

Principle #15Dynamics

2Productivity

If non-autoregressive (NAR) models are used for data prediction, then processing efficiency is improved, but prediction accuracy decreases

Engineering Contradiction:

Improvetoken generation efficiencyVSAvoidprediction accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The patent segments the sequence generation process into two distinct phases: an autoregressive phase for generating initial tokens with high accuracy, and a non-autoregressive phase for generating remaining tokens in parallel. This segmentation allows the system to apply different processing strategies to different parts of the sequence, optimizing both accuracy and efficiency.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Different processing qualities are applied to different parts of the sequence: high-quality autoregressive processing for initial tokens that require accuracy, and efficient non-autoregressive processing for subsequent tokens where speed is more critical. This local differentiation of processing quality optimizes the overall system performance.

Inventive Principle:

Principle #3Local quality

3Measurement precision

If autoregressive models are deployed for high-volume data processing, then accuracy is maintained, but system latency becomes unacceptably long

Engineering Contradiction:

Improveprediction accuracyVSAvoidsystem latency

Core Design Contradiction:

Measurement precisionVSSpeed

Solution Approach 1:

The patent segments the sequence generation process into two distinct phases: an autoregressive phase for generating initial tokens with high accuracy, and a non-autoregressive phase for generating remaining tokens in parallel. This segmentation allows the system to apply different processing strategies to different parts of the sequence, optimizing both accuracy and efficiency.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system employs periodic switching between processing modes, starting with autoregressive processing for the initial period (first n tokens) and then transitioning to non-autoregressive processing for the remainder. This periodic action pattern allows the system to satisfy both accuracy requirements for critical initial tokens and latency requirements for overall processing speed.

Inventive Principle:

Principle #19Periodic action

Data Source

PatentUS12530528B2Generation of data models for predicting data

Publication Date: 2026.01.20 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

Systems and methods are provided for training a data model based on training data. The training includes pre-training and fine-tuning the data model based on a combination of an autoregressive (AR) model and a non-autoregressive (NAR) model. Training data may be received and encoded into streams of tokens. A pre-trainer during decoding generates a continuum of data structures of the AR and NAR combined model including a main stream and a series of predicting streams. Masked tokens in predicting streams reference or attend to one or more preceding tokens in the main stream or the preceding predicting streams. A fine-tuner selects streams to generate a trained model according to a target data model. The target data model is determined based on balancing an accuracy constraint and an efficiency constraint for predicting tokens. The decoder acts as abridge between the AR and NAR models in generating a trained data model.