Natural Language Processing Apparatus for Unknown Word Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing natural language processing methods fail to efficiently identify unknown words by simply categorizing all unmatched dictionary words as unknown, leading to inefficient word extraction.
Innovation Solution
A natural language processing apparatus and method that utilize multiple morphological analyzers to detect common segmentation points and extract part-of-speech differing patterns, storing these patterns with their frequency, and specifying character strings with low frequency appearances as candidates for unknown words, thereby limiting the search range for unknown words.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all words that do not match any words in a dictionary are determined as unknown words, then unknown words can be identified, but the identification process becomes inefficient and produces many false positives
Solution Approach 1:
The patent segments the text processing task by dividing it into multiple stages: initial morphological analysis by multiple analyzers, extraction of differing patterns, frequency counting, and threshold-based filtering. This segmentation allows the system to efficiently narrow down candidate unknown words without examining every single unmatched word, thus improving both accuracy and efficiency.
Solution Approach 2:
Instead of performing complete morphological analysis on all words using multiple analyzers and then filtering, the patent applies partial action by first using multiple analyzers to identify differing patterns, then using frequency counting and thresholding to process only the most likely candidates. This reduces the overall computational burden while maintaining detection accuracy.
2Measurement precision
If multiple morphological analyzers are used to analyze sentences, then the accuracy of unknown word detection improves, but the computational complexity and processing time increase
Solution Approach 1:
The patent extracts only the essential information needed for unknown word detection by comparing the output of multiple morphological analyzers. Specifically, it extracts differing patterns between analyzers and counts their frequencies, discarding redundant analysis results. This extraction approach maintains high detection accuracy while reducing computational complexity.
Solution Approach 2:
The patent changes the parameter of analysis by using multiple morphological analyzers with different dictionaries or analysis rules, then compares their outputs to identify differing patterns. By transforming the problem from complete analysis of all words to comparison of analyzer differences, the system achieves higher accuracy without proportional increases in complexity.
3Measurement precision
If the search range for unknown words is not limited, then all potential unknown words can be detected, but the processing time and computational resources increase significantly
Solution Approach 1:
The patent performs preliminary action by using multiple morphological analyzers to pre-process the text and identify differing patterns before the actual unknown word detection. By counting the frequencies of these differing patterns and applying a threshold, the system pre-filters candidate words, significantly reducing the search range and processing time while maintaining detection completeness.
Solution Approach 2:
The patent substitutes the mechanical approach of examining every single word against a dictionary with a more efficient system based on pattern recognition and frequency analysis. By replacing brute-force comparison with statistical pattern matching, the system achieves comprehensive detection with reduced time loss.
Data Source
AI summary
A natural language processing apparatus includes a result acquisition unit that acquires a plurality of analysis results indicating parts of speech of morphemes contained in one or more common sentences from a plurality of types of morphological analyzers, a pattern acquisition unit that detects a common segmentation point in the plurality of analysis results, extracts one or more parts of speech corresponding to a character string segmented at the common segmentation point from each of the analysis results, and acquires a set of the parts of speech as a part-of-speech differing pattern, and a candidate specifying unit that extracts the part-of-speech differing pattern with the number of appearances being equal to or less than a predetermined threshold and specifies the character string corresponding to the extracted part-of-speech differing pattern as a character string containing a candidate for an unknown word.