Neural Network Training for Unknown Words via Character N-grams
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods are unable to effectively train neural networks to calculate word scores for new words not registered in known-word dictionaries.
Innovation Solution
A training apparatus that includes a dividing unit, a converting unit, and a calculating unit, which divides input character strings into words, converts them into vocabulary features based on probabilities, and trains the neural network using the difference between expected and actual output scores.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional training methods using known-word dictionaries are used, then the neural network can calculate word scores for registered words, but it cannot effectively handle new words not in the dictionary
Solution Approach 1:
The patent introduces character n-grams as an intermediary representation between input words and the neural network. Instead of directly using whole words (which require dictionary registration), the system breaks down words into character sequences of length n, which can be processed even for unseen words. This intermediary representation enables the network to handle new words while maintaining reliable score calculations through character-level pattern recognition.
Solution Approach 2:
The patent changes the parameter representation from word-level to character-n-gram level. By transforming the input from complete words to subsequences of characters, the system alters the fundamental parameter structure, allowing the neural network to process unfamiliar words through their character compositions rather than requiring exact word matches in the dictionary.
2Adaptability or versatility
If the neural network is trained with character n-grams to handle new words, then adaptability to unknown words improves, but the computational complexity and training data processing increases
Solution Approach 1:
The patent segments input words into multiple overlapping character n-grams. Each word is broken down into all possible contiguous subsequences of length n, creating a set of smaller units that can be independently processed. This segmentation reduces the complexity of handling entire unknown words by decomposing them into manageable character sequences that can be efficiently trained and processed.
Solution Approach 2:
The patent performs preliminary processing to generate all character n-grams from the training corpus before neural network training. By pre-computing and storing character n-gram representations along with their positional information, the system prepares the training data in advance, reducing the computational burden during the actual training phase and enabling efficient processing of the segmented data.
3Manufacturing precision
If morphological analysis is performed on all training texts to convert to word strings, then the training data quality improves, but the processing time and computational resources increase
Solution Approach 1:
The patent extracts only the necessary character n-gram information from the training texts without performing complete morphological analysis on all texts. By selectively extracting character sequences of length n and their positional relationships, the system obtains sufficient training data quality while avoiding the time-consuming process of full morphological analysis, focusing only on the essential features needed for the task.
Data Source
AI summary
According to an embodiment, a training apparatus includes a converting unit to convert each input word into a vocabulary feature thereof based on a first probability associated with the input word, to acquire an input training word indicating the input word or the vocabulary feature, and convert, when an input training word string including input training words is input to an input layer of a neural network, an output word expected to be output from an output layer of the neural network into a vocabulary feature of the output word based on a second probability associated with the output word, to acquire an output training word indicating the output word or a vocabulary feature thereof; and a training unit to train the neural network based on a difference between an expected score of the output training word and a score output from a unit corresponding to the output training word.