Morphological Analysis Index for CJK Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing morphological analysis techniques for CJK characters struggle to efficiently segment character strings into smallest meaningful units at high speed with reduced file size, leading to inadequate performance in applications like Word2Vec, where increased dictionary entries result in larger Trie and Double-Array structures and longer retrieval times.
Innovation Solution
A computer-readable medium stores an analysis program that generates an index using a morphological dictionary to identify the beginning and end of each morpheme, allowing for efficient extraction of dividable words from input character data by setting flags at specific offsets, enabling high-speed analysis with reduced file size.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If more entries are included in the morphological dictionary to extract more dividable word candidates, then the accuracy of morpheme segmentation is improved, but the file size of Trie and Double-Array increases and retrieval time becomes longer
Solution Approach 1:
The patent segments the morphological dictionary into multiple smaller dictionaries organized by character. Instead of using a single large Trie or Double-Array structure, it divides the dictionary into character-specific sub-dictionaries, allowing faster retrieval while maintaining comprehensive word coverage. This segmentation reduces the time complexity of word lookup operations.
Solution Approach 2:
The patent introduces a new dimensional approach by organizing the morphological dictionary along the character dimension rather than using traditional single-structure methods. It creates a multi-dimensional indexing system where words are accessed through their constituent characters, transforming the one-dimensional search problem into a multi-dimensional lookup process that improves efficiency.
2Measurement precision
If more entries are included in the morphological dictionary to extract more dividable word candidates, then the accuracy of morpheme segmentation is improved, but the file size of Trie and Double-Array increases
Solution Approach 1:
The patent segments the morphological dictionary into multiple smaller dictionaries organized by character. Instead of using a single large Trie or Double-Array structure, it divides the dictionary into character-specific sub-dictionaries, allowing faster retrieval while maintaining comprehensive word coverage. This segmentation reduces the time complexity of word lookup operations.
Solution Approach 2:
The patent extracts only the necessary components from the traditional morphological dictionary structure. It extracts character-based indexing information and essential word boundary flags, discarding redundant data structures. This extraction approach maintains segmentation accuracy while significantly reducing file size.
3Measurement precision
If traditional morphological analysis methods are used to segment CJK character strings, then word boundaries can be identified, but the analysis speed is insufficient for high-performance applications
Solution Approach 1:
The patent performs preliminary action by pre-processing and organizing the morphological dictionary into character-based sub-dictionaries with pre-computed word boundary flags. This preliminary organization allows the analysis system to quickly retrieve and process word boundary information during runtime, significantly improving analysis speed without sacrificing accuracy.
Solution Approach 2:
The patent implements self-service through automatic word boundary detection using pre-computed flags in the character-based dictionaries. The system automatically identifies word boundaries by checking these flags during character sequence processing, eliminating the need for complex real-time analysis algorithms and thereby improving processing speed.
Data Source
AI summary
A non-transitory computer-readable recording medium stores therein an analysis program that causes a computer to execute a process including: generating an index based on a dictionary for use in morphological analysis, the index relating to morphemes registered in the dictionary, the index including flags by which a beginning and an end of each morpheme registered in the dictionary are determinable; and extracting a plurality of dividable words from input character data by using the index.