Token Sequence Folding for Large Language Model Training

Solution Overview

Problem

Large language models face significant computational and memory costs due to their increasing complexity with input length, limiting their ability to efficiently handle long-text tasks and requiring substantial resources for training and deployment.

Innovation Solution

A method involving token sequence folding, where an initial token sequence is compressed using a folding feature value to create a shorter first token sequence, which is then input into a model for training, reducing computational complexity and memory footprints, and improving model efficiency for long-text tasks.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If large models are trained on massive textual data to achieve general semantic understanding and multi-task learning capabilities, then the model demonstrates strong generalization capabilities, but the computational consumption and video memory footprints increase in square level with the length of the inputs, bringing significant cost overhead for training and deploying models

Engineering Contradiction:

Improvegeneralization capabilitiesVSAvoidcomputational consumption

Core Design Contradiction:

Adaptability or versatilityVSUse of energy by moving object

Solution Approach 1:

The patent segments the token sequence into multiple segments and processes them through folding operations. Instead of processing the entire long sequence at once, the model divides it into manageable segments that can be processed efficiently, reducing the computational burden while maintaining the ability to handle long-text tasks

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces a folding dimension to transform the sequential processing of tokens into a multi-dimensional structure. By folding the token sequence, the model creates a new dimensional representation that reduces the linear sequence length while preserving semantic information, effectively changing from a 1D sequence problem to a multi-dimensional transformation problem

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

2Adaptability or versatility

If large models are trained on massive textual data to achieve general semantic understanding and multi-task learning capabilities, then the model demonstrates strong generalization capabilities, but the video memory footprints increase in square level with the length of the inputs, bringing significant cost overhead for training and deploying models

Engineering Contradiction:

Improvegeneralization capabilitiesVSAvoidvideo memory footprints

Core Design Contradiction:

Adaptability or versatilityVSVolume of stationary object

Solution Approach 1:

The patent segments the token sequence into multiple segments and processes them through folding operations. Instead of loading the entire long sequence into memory at once, the model divides it into manageable segments that can be processed efficiently, reducing the video memory footprint while maintaining the ability to handle long-text tasks

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces a folding dimension to transform the sequential processing of tokens into a multi-dimensional structure. By folding the token sequence, the model creates a new dimensional representation that reduces the linear sequence length while preserving semantic information, effectively changing from a 1D sequence problem to a multi-dimensional transformation problem

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

3Adaptability or versatility

If large models are trained on massive textual data to achieve general semantic understanding and multi-task learning capabilities, then the model demonstrates strong generalization capabilities, but this limits their capabilities to solve long-text tasks due to the significant cost overhead

Engineering Contradiction:

Improvegeneralization capabilitiesVSAvoidcapability to solve long-text tasks

Core Design Contradiction:

Adaptability or versatilityVSProductivity

Solution Approach 1:

The patent segments the token sequence into multiple segments and processes them through folding operations. This segmentation enables the model to efficiently process long-text tasks by breaking them into manageable pieces that can be handled within available computational resources, thereby improving productivity on long-text tasks while maintaining generalization capabilities

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces a folding dimension to transform the sequential processing of tokens into a multi-dimensional structure. By folding the token sequence, the model creates a new dimensional representation that reduces the linear sequence length while preserving semantic information, enabling efficient processing of long-text tasks

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

Data Source

PatentUS20250094802A1Model training method, model reasoning method, electronic device, and storage medium

Publication Date: 2025.03.20 BEIJING BAIDU NETCOM SCI & TECH CO LTD

AI summary

Provided is a model training method, a model reasoning method, an electronic device, and a storage medium, relating to the field of data processing, and especially to the technical fields of artificial intelligence, big data, deep learning and large models. The model training method includes: folding an initial token sequence for training a model based on a folding feature value for folding a token sequence to obtain at least a first token sequence subjected to the folding, wherein the initial token sequence represents a token sequence composed of T1 tokens, and the first token sequence has a sequence length less than that of the initial token sequence; and inputting at least the first token sequence into a preset model to train the preset model so as to obtain a target model.