Neural Sequence Modeling via Output Segmentations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing sequence modeling approaches fail to effectively leverage segmental structure in natural-language sequences, leading to suboptimal output predictions in tasks like machine translation and speech recognition, where the length and alignment of input and output sequences can vary significantly.
Innovation Solution
A neural-network-based approach that models output sequences as a product of segment probabilities, allowing for output sequences of any length, with each input element emitting exactly one output segment, and using recurrent neural networks to condition on both input and preceding output segments, enabling incremental predictions through a beam search algorithm.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If sequential mapping from input to output elements is used, then the mapping process is simple, but the prediction accuracy deteriorates when word order is not preserved or word counts differ
Solution Approach 1:
The patent segments the output sequence into multiple segments, where each input element can emit one or more segments. This segmentation allows the model to capture phrase-level structures and contextual relationships while maintaining a manageable computational approach, resolving the contradiction between simple mapping and accurate prediction.
2Manufacturing precision
If the output sequence is conditioned on the entire input sequence, then the prediction accuracy improves, but the ability to make incremental predictions deteriorates
Solution Approach 1:
The patent introduces dynamic segmentation where the output sequence is divided into variable-length segments that can be emitted incrementally. Each input element dynamically determines which segments to emit based on the current context, allowing the model to maintain accuracy while enabling incremental prediction capabilities.
3Device complexity
If fixed segment length is used, then the computational complexity is reduced, but the ability to model variable-length phrases deteriorates
Solution Approach 1:
The patent changes the parameter of segment length from fixed to variable, allowing segments to adapt to the natural boundaries of phrases and linguistic structures. This enables the model to capture variable-length phrases effectively while managing computational complexity through the segment-based approach.
Data Source
AI summary
In neural-network-based approaches to sequence modeling, an output sequence may be modeled via segmentations, the probability of the output sequence being constructed as a sum of products of output-segment probabilities, taken over all valid output-sequence segmentations. A set of artificial neural networks may model the distribution of the output-sequence probability with a recurrent neural network modeling the distributions of the individual output-segment probabilities, optionally in conjunction with a second recurrent neural network modeling concatenations of output segments. In various embodiments, this approach is applied to neural phrase-based machine translation.