Transformer Token Compression Using Merged Position Values
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformer models require large amounts of hardware resources due to lengthy training data sequences, which slows down convergence and efficiency.
Innovation Solution
Compressing tokens based on their positions by identifying and combining position values for identical tokens, generating training data with shorter sequences, and using embedding spaces to reduce dimensionality, thereby training a smaller neural network.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If transformer models use lengthy training data sequences, then the model can capture more context information, but the hardware resources required increase significantly
Solution Approach 1:
The patent merges identical tokens by combining their position values into a single representative position. For example, if the token 'the' appears at positions 1, 5, and 10, it is replaced with a single occurrence at position 1 (the first position), and the position values 5 and 10 are combined with position 1. This reduces the sequence length while preserving the contextual information that identical tokens provide about the structure and distribution of language.
2Loss of information
If transformer models process lengthy sequences, then the model achieves better understanding, but convergence speed decreases
Solution Approach 1:
By merging identical tokens and their position values, the patent reduces the effective sequence length processed by the transformer. This compression maintains the linguistic information needed for understanding while reducing the computational burden per training step, thereby accelerating convergence without sacrificing language understanding capability.
3Use of energy by moving object
If the training data sequence length is reduced, then hardware requirements decrease, but the model may lose detailed positional information
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing position embeddings for all possible positions in the vocabulary before training. These pre-computed position embeddings capture positional information in advance, allowing the model to reconstruct accurate positional relationships even after token compression. The position embeddings are retrieved and applied during training based on the compressed position indices.
Data Source
AI summary
Embodiments of the present disclosure include systems and methods for compressing tokens based on positions for training data that is used to train transformer models. In some embodiments, a set of input data for training a transformer model is received. The set of input data comprises a set of tokens and a set of position values. A first token in the set of tokens that is the same as a second token in the set of tokens is identified. The position value representing the first token with the position value representing the second token are combined. The set of tokens is modified by removing the first token from the set of tokens. A set of training data is generated to comprise the modified set of tokens and the set of position values. The transformer model is trained using the set of training data.