Fixed-Length Token Packing for Faster Sequence Model Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for training sequence models are inefficient, leading to slow convergence of weights and prolonged training times.
Innovation Solution
Implementing token packing techniques such as repeating, stacking, and optimizing methods to generate input data for sequence models, which involve grouping tokens and filling fixed-length data structures efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional token processing methods are used for training sequence models, then the training process is simple to implement, but the training time is prolonged and weight convergence is slow
Solution Approach 1:
The patent segments the fixed-length data structure into multiple token slots that can be independently filled with different tokens from the input sequence. This segmentation allows parallel processing of multiple tokens simultaneously, accelerating the training process while maintaining compatibility with standard sequence model architectures.
Solution Approach 2:
The patent performs preliminary token packing into fixed-length data structures before feeding data to the sequence model. By pre-organizing tokens into structured formats with proper positioning and padding, the system eliminates the need for complex runtime processing, thereby reducing training time and improving computational efficiency.
2Productivity
If token packing techniques are implemented to accelerate training, then weight convergence speed increases, but the data processing complexity increases
Solution Approach 1:
The patent changes the parameter representation by introducing position indicators and padding tokens within the fixed-length data structure. These parameter changes enable the model to distinguish between actual tokens and filler positions, maintaining information integrity while accelerating training through efficient data packing.
Solution Approach 2:
The patent introduces an intermediary token packing layer that transforms raw input sequences into structured fixed-length data structures. This intermediary layer handles the complexity of token arrangement, positioning, and padding, thereby simplifying the subsequent model processing while maintaining high convergence speed.
3Quantity of substance
If fixed-length data structures are used for token packing, then memory efficiency improves, but information loss may occur due to truncation
Solution Approach 1:
The patent uses special padding tokens to fill unused positions in fixed-length data structures. These padding tokens act as copies or placeholders that preserve the structural integrity of the data without introducing actual information loss, allowing the model to distinguish between meaningful tokens and filler positions.
Solution Approach 2:
The patent nests multiple levels of structure within the fixed-length data structure, including token sequences, position indicators, and padding markers. This nested organization allows efficient memory utilization while preserving complete information about the original input sequence through hierarchical encoding.
Data Source
AI summary
Embodiments of the present disclosure include systems and methods for packing tokens to train sequence models. In some embodiments, a plurality of datasets for training a sequence model is received. Each dataset in the plurality of datasets includes a sequence of correlated tokens. A set of training data is generated that includes a subset of a sequence of tokens from a first dataset in the plurality of datasets and a subset of a sequence of tokens from a second, different dataset in the plurality of datasets. The sequence model is trained using the set of training data.