Neural Machine Translation Rare Word Processing via Pointer Tokens
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural machine translation (NMT) systems are limited by a fixed and modest-size vocabulary, making them incapable of translating rare words and relying on a single symbol to represent out-of-vocabulary words, leading to poor translation performance for sentences with many rare words.
Innovation Solution
A system that includes a neural network translation model, a rare word processing subsystem, and a word dictionary, which trains to emit pointer tokens and null unknown tokens to track the origin of unknown words in source sentences, and uses a dictionary to replace these tokens with corresponding source words, employing alignment data and annotation strategies like the copyable, positional all, and positional unknown models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If a fixed and modest-size vocabulary is used in NMT systems, then the system is simple to implement and train, but it becomes incapable of translating rare words and sentences with many rare words are translated poorly
Solution Approach 1:
The patent segments the translation process into two distinct components: a neural network translation model that handles common vocabulary translation, and a separate rare word processing subsystem that specifically addresses out-of-vocabulary words. This segmentation allows each component to be optimized for its specific function while working together to solve the overall translation problem, resolving the contradiction between system simplicity and rare word handling capability
Solution Approach 2:
The patent introduces alignment data as an intermediary element that connects the source sentence and target sentence. This alignment information serves as a mediator that enables the rare word processing subsystem to identify and translate rare words by matching them with corresponding entries in the word dictionary, without requiring the neural network model itself to be expanded or modified
2Quantity of substance
If a single symbol is used to represent all out of vocabulary words, then the vocabulary size is kept manageable, but translation quality for sentences with many rare words deteriorates
Solution Approach 1:
The patent implements a dynamic vocabulary representation system where the effective vocabulary size expands when rare words are encountered. The system maintains a manageable base vocabulary in the neural network model, but dynamically accesses additional word entries from the word dictionary through the rare word processing subsystem, allowing the system to adapt its vocabulary coverage to the specific translation needs without permanently increasing system complexity
Solution Approach 2:
The patent performs preliminary processing by generating alignment data between source and target sentences before the actual translation is finalized. This alignment information is prepared in advance and used by the rare word processing subsystem to identify rare words that need translation, enabling the system to handle out-of-vocabulary words efficiently without compromising translation quality
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for neural translation systems with rare word processing. One of the methods is a method training a neural network translation system to track the source in source sentences of unknown words in target sentences, in a source language and a target language, respectively and includes deriving alignment data from a parallel corpus, the alignment data identifying, in each pair of source and target language sentences in the parallel corpus, aligned source and target words; annotating the sentences in the parallel corpus according to the alignment data and a rare word model to generate a training dataset of paired source and target language sentences; and training a neural network translation model on the training dataset.