Unknown-Word Embedding for More Accurate Sequence Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural machine translation systems struggle with generating unknown words, as subword units do not account for learning embedded vectors for rare or unseen words, leading to inaccurate predictions.
Innovation Solution
An information learning apparatus generates third embedded vectors for processing units using subword units and adjusts learning parameters based on input sequences to improve generation accuracy, incorporating a pre-processing unit for subword division and hierarchical vector representation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If subword units are used to handle unknown words, then the theoretical number of generatable words is increased, but embedded vectors for unknown words cannot be learned
Solution Approach 1:
The patent introduces a character embedding layer that is initialized with random vectors before training. This preliminary initialization allows the system to handle characters not seen in training data by using the random initialization as a starting point, enabling the model to learn appropriate embeddings during training without requiring explicit pre-learning of all possible characters
Solution Approach 2:
The patent uses character-level embeddings as an intermediary representation between the input sequence and the final output. By breaking down words into characters and learning character embeddings, the system can compose meaningful representations for unknown words through combination of character-level information, bridging the gap between subword units and learnable embeddings
2Ease of manufacture
If rare words are replaced with unknown words, then the training data is simplified, but generation accuracy for rare words deteriorates
Solution Approach 1:
The patent segments words into character-level units and processes them individually through the embedding layer. This segmentation allows the system to handle rare words by processing their constituent characters, enabling the model to reconstruct rare words accurately through combination of character embeddings rather than requiring the entire rare word to appear in training data
Solution Approach 2:
The patent changes the embedding approach from word-level to character-level embeddings. By using character embeddings with random initialization and training them to represent character-level information, the system can accurately represent rare words through the combination of character embeddings, improving generation accuracy without requiring rare words to appear frequently in training data
Data Source
AI summary
An information learning apparatus includes a memory and a processor configured to perform generating, for each of processing units constituting an input sequence included in training data, a third embedded vector based on a first embedded vector for the processing unit and a second embedded vector corresponding to an unknown word; executing a process based on a learning target parameter, with the third embedded vector generated for each of the processing units as an input; and learning, for a processing result by the executing, the parameter based on an error of an output corresponding to the input sequence in the training data.