Self-Attention Model Position Information Fusion for OOV Handling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In natural language processing, self-attention models face challenges with out-of-vocabulary words due to smaller vocabulary lists, and there is a need to incorporate word position information to improve model accuracy.
Innovation Solution
A method and apparatus for fusing position information by segmenting words into subwords and generating absolute position information, which is then used to enhance the self-attention model for more accurate training and prediction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a smaller vocabulary list is used in self-attention models, then model complexity is reduced, but out-of-vocabulary problems worsen
Solution Approach 1:
The patent segments words into subwords (word pieces) to create a hierarchical representation system. This allows the model to handle both common words (as complete units) and rare/unknown words (as combinations of subwords), resolving the contradiction between vocabulary size and OOV handling capability
Solution Approach 2:
The patent creates a composite representation system that combines word-level embeddings and subword-level embeddings. This composite approach allows the model to leverage both complete word representations for known words and compositional subword representations for unknown words, simultaneously achieving small vocabulary size and good OOV handling
2Reliability
If subword segmentation is applied, then out-of-vocabulary problems are improved, but position information accuracy deteriorates
Solution Approach 1:
The patent implements a nested structure where subwords are nested within words, and position information is nested at multiple levels. Each subword inherits the position information of its parent word, creating a hierarchical position representation that maintains both subword granularity and word-level position accuracy
Solution Approach 2:
The patent adds a hierarchical dimension to position representation by introducing both word-level positions and subword-level positions. This multi-dimensional position system allows the model to capture both the position of complete words and the relative positions of subwords within those words, resolving the precision loss from segmentation
Data Source
AI summary
A method and an apparatus for fusing position information, and a non-transitory computer-readable recording medium are provided. In the method, words of an input sentence are segmented to obtain a first sequence of words in the input sentence, and absolute position information of the words in the first sequence is generated. Then, subwords of the words in the first sequence are segmented to obtain a second sequence including subwords, and position information of the subwords in the second sequence are generated, based on the absolute position information of the words in the first sequence, to which the respective subwords belong. Then, the position information of the subwords in the second sequence are fused into a self-attention model to perform model training or model prediction.