Neural Network Token Segmentation for Sequence Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network training methods for generating output sequences are limited in accuracy and computational efficiency, particularly in tasks like translation and speech recognition, as they assume a fixed output sequence decomposition and are computationally expensive due to long decoder lengths.
Innovation Solution
A method for training neural networks to generate score distributions over sets of tokens including n-grams of multiple sizes, where the networks process inputs to adjust parameters based on valid decompositions of target sequences, allowing for more accurate and efficient generation of output sequences by conditioning on network inputs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If neural networks use fixed output sequence decomposition, then training is simpler, but accuracy is limited
Solution Approach 1:
The patent applies dynamics by transforming the fixed decomposition approach into a dynamic one where the decomposition is learned adaptively during training. The network learns to decompose output sequences into variable-length tokens based on the input context, allowing the decomposition structure to change and optimize for each specific translation task rather than being predetermined.
Solution Approach 2:
The patent applies segmentation by dividing the output sequence into multiple variable-length tokens (subwords, characters, or complete words) that can be dynamically determined during training. This segmentation allows the network to learn optimal token boundaries and representations, improving accuracy while managing complexity through structured tokenization.
2Measurement precision
If neural networks process long decoder sequences, then complete output sequences are generated, but computational cost increases
Solution Approach 1:
The patent applies segmentation by dividing long decoder sequences into shorter variable-length tokens. Instead of processing entire long sequences as single units, the network processes segmented tokens that can be dynamically adjusted in length, reducing the computational burden while maintaining the ability to generate complete output sequences.
Solution Approach 2:
The patent applies dynamics by making token lengths adaptive rather than fixed. The network learns to adjust token lengths dynamically based on the input sequence and translation requirements, allowing shorter tokens for complex regions and longer tokens for simpler regions, thereby optimizing computational efficiency while maintaining output quality.
3Measurement precision
If neural networks use variable token lengths, then accuracy improves, but training complexity increases
Solution Approach 1:
The patent applies segmentation by introducing a structured tokenization system that divides output sequences into variable-length tokens with defined boundaries and representations. This structured approach to segmentation provides a framework that manages training complexity while enabling the network to learn optimal variable-length token representations for improved translation accuracy.
4Productivity
If neural networks generate sequences quickly, then productivity improves, but accuracy may decrease
Solution Approach 1:
The patent applies dynamics by making the tokenization strategy adaptive and context-dependent. The network learns to dynamically adjust token lengths and boundaries based on the specific translation requirements, allowing it to generate sequences quickly for simple cases while maintaining high accuracy for complex cases by using more granular token representations when needed.
Data Source
AI summary
A computer-implemented method for training a neural network that is configured to generate a score distribution over a set of multiple output positions. The neural network is configured to process a network input to generate a respective score distribution for each of a plurality of output positions including a respective score for each token in a predetermined set of tokens that includes n-grams of multiple different sizes. Example methods described herein provide trained neural networks which produce results with improved accuracy compared to the state of the art, e.g. translations that are more accurate compared to the state of the art, or more accurate speech recognition compared to the state of the art.