Morpheme Analysis Learning Device for Unknown Word Adaptability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional Japanese morphological analysis methods struggle with high accuracy when texts contain many unknown words not registered in the morpheme dictionary, leading to a tradeoff between analysis accuracy for known and unknown words, and the coverage ratio of the morpheme dictionary is often unknown or variable.
Innovation Solution
A morphological analysis learning apparatus that divides input text into character units, acquires character and morpheme vectors, and combines them with a specific probability to learn a series predictor, allowing for high analysis accuracy irrespective of the morpheme dictionary's coverage ratio by artificially simulating both high and low coverage scenarios during training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a morpheme dictionary-based analysis method is used, then analysis accuracy is high for text with high coverage ratio, but analysis accuracy deteriorates for text with many unknown words (low coverage ratio)
Solution Approach 1:
The patent segments the input text into character units and processes each character independently to generate character vectors. This segmentation allows the model to handle unknown words by analyzing individual characters rather than relying on complete word matches in the morpheme dictionary, thereby resolving the contradiction between dictionary-based accuracy and unknown word adaptability.
Solution Approach 2:
The patent changes the parameter of vector combination by controlling the combination probability parameter. During training, this parameter is adjusted to simulate different morpheme dictionary coverage scenarios (high coverage when combination probability is high, low coverage when combination probability is low). This enables the model to learn adaptively across varying dictionary coverage conditions, resolving the contradiction between optimization for high coverage and robustness for low coverage.
2Measurement precision
If the morpheme dictionary coverage ratio is optimized for high coverage text, then analysis accuracy is high for standard text, but analysis accuracy deteriorates for text with many colloquial expressions and errors
Solution Approach 1:
The patent performs preliminary action by pre-training the model with simulated varying coverage ratios before actual analysis. During training, the model is exposed to both high coverage and low coverage scenarios through controlled vector combination probabilities, preparing it to handle diverse text types including colloquial expressions and erroneous text, while maintaining accuracy for standard text.
3Adaptability or versatility
If a character-unit analysis method is used, then adaptability to unknown words is improved, but analysis accuracy deteriorates for text with many words registered in the morpheme dictionary
Solution Approach 1:
The patent merges two analysis approaches by combining character-unit analysis with morpheme dictionary-based analysis. The system generates both character vectors (for unknown word adaptability) and morpheme vectors (for dictionary word accuracy), then combines them with a controllable probability. This merging resolves the contradiction by allowing the model to leverage both methods' strengths simultaneously.
Solution Approach 2:
The patent uses parameter changes by adjusting the vector combination probability during training to simulate different morpheme dictionary coverage scenarios. This enables the model to learn when to rely on character-unit analysis (for unknown words) and when to rely on morpheme dictionary matching (for registered words), resolving the contradiction between adaptability and accuracy.
Data Source
AI summary
It is possible to achieve high analysis accuracy on the whole irrespective of a coverage ratio of a morpheme dictionary. A morpheme prediction section 38 outputs a morpheme label string as a morphological analysis result of input text by inputting a resultant vector generated for each character by a vector combination section 36 into a series predictor and sequentially performing calculation. At this point, there is used the series predictor which is subjected to learning in advance such that a morpheme label string obtained by inputting a resultant vector in which a character vector and a morpheme information vector are combined and which is generated without being combined with the morpheme information vector with a specific probability into the series predictor and sequentially performing the calculation coincides with a morpheme label string of learning data.