Token Position Handling for Transformer Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network training methods are inefficient due to the need to process large datasets, including redundant and common tokens, which slows down the training process and hinders the learning of rare tokens.
Innovation Solution
The proposed solution involves token dropping, masking, and reordering techniques, where common tokens are removed or masked, and their positions are reordered, allowing the neural network to focus on less frequent tokens, thereby reducing the input size and accelerating training by using orthogonal token representations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If large datasets including all tokens are processed, then the training data completeness is improved, but the training time increases and convergence slows down
Solution Approach 1:
The patent extracts and removes common tokens from the training dataset, keeping only rare tokens for training. This extraction principle directly addresses the contradiction by removing unnecessary data elements (common tokens) that consume training time without contributing meaningfully to model performance, thus reducing training time while maintaining data completeness for important rare tokens
Solution Approach 2:
The patent applies different treatment to different tokens based on their frequency characteristics. Rare tokens are preserved and emphasized in training, while common tokens are removed. This local quality approach allows the training process to focus computational resources on the most valuable data points, resolving the contradiction between data completeness and training efficiency
2Adaptability or versatility
If all tokens are included in training, then the model learns comprehensive language patterns, but the network cannot focus on rare tokens effectively
Solution Approach 1:
By extracting and removing common tokens from the training set, the patent enables the network to focus exclusively on rare tokens. This extraction ensures that rare tokens receive sufficient attention and computational resources during training, improving learning effectiveness for these important but infrequent elements
Solution Approach 2:
The patent applies selective attention by treating rare tokens differently from common tokens. The training process is optimized to give priority coverage to rare tokens through techniques like token dropping and selective masking, ensuring these important tokens are learned effectively while maintaining adaptability through preserved language patterns
3Stability of the object's composition
If common tokens are included in training sequences, then the input sequence maintains natural language structure, but the training efficiency decreases due to redundant information
Solution Approach 1:
The patent extracts and removes common tokens from training sequences, eliminating redundant information that reduces training efficiency. This extraction maintains language structure integrity by preserving the positions and contextual relationships of rare tokens while removing only the redundant common elements
Solution Approach 2:
The patent discards common tokens during training but recovers their positional information through specialized embedding techniques. This allows the model to learn from rare tokens efficiently while still understanding the structural context where common tokens would normally appear, maintaining language structure without the efficiency penalty
Data Source
AI summary
Embodiments of the present disclosure include a method for token-position handling comprising: processing a first sequence of tokens to produce a second sequence of tokens, wherein the second sequence of tokens has a smaller number of tokens than the first sequence of tokens; masking at least some tokens in the second sequence to produce masked tokens; moving the masked tokens to the beginning of the second sequence to produce a third sequence; encoding tokens in the third sequence into a set of numeric vectors in a first array; and processing the first array in a transformer neural network to determine correlations among the third sequence, the processing the first array producing a second array.