Neural Machine Translation Word Vector Update via Head Representation Feedback
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional neural machine translation models, such as the Transformer, do not fully utilize semantic information among head representations at the same and different times, limiting their translation performance.
Innovation Solution
A method and apparatus for processing word vectors in neural machine translation models by dividing each word vector into multiple head representations, calculating weighted sums based on correlation and attention parameters, and combining these representations to update the word vectors, thereby enhancing semantic information utilization during model training and translation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the Transformer model uses a multi-head attention mechanism to extract semantic feature information, then the translation performance is improved, but the semantic information utilization among head representations is insufficient
Solution Approach 1:
The patent introduces a feedback mechanism where the attention output is fed back to update the word vectors. The attention mechanism calculates attention scores between query and key vectors, generates attention outputs, and these outputs are used to update the original word vectors through a combination of the attention output and the original word vector. This feedback loop ensures that semantic information is fully utilized and retained in the updated word vectors, resolving the issue of insufficient semantic information utilization while maintaining improved translation performance.
2Adaptability or versatility
If word vectors are divided into multiple head representations for multi-head attention, then the model captures diverse semantic features, but the correlation among head representations is not fully utilized
Solution Approach 1:
The patent merges the attention outputs from multiple head representations with the original word vectors to create updated word vectors. The merging process combines the diverse semantic features captured by different heads while preserving the correlations among them. By concatenating or summing the attention outputs with the original word vectors, the model integrates information from all head representations, ensuring that correlation information is not lost and diverse semantic features are fully utilized.
Data Source
AI summary
A method and an apparatus for processing word vectors of a neural machine translation model, and a non-transitory computer-readable recording medium are provided. In the method, word vectors that are input to an encoder and a decoder of a neural machine translation model are updated using semantic information among head representations at the same time and semantic information among head representations at different times, and the model is trained or translation is performed using the updated word vectors, thereby improving the model performance of the neural machine translation model.