Hybrid N-gram Lexical Text Segmentation for Asian Languages
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current text analysis methods for non-Indo-European languages like Chinese and Japanese face issues with incomplete lexicons in lexical analysis and spurious results in n-gram tokenization, leading to suboptimal search and classification outcomes.
Innovation Solution
A hybrid n-gram/lexical analysis tokenization system that performs both tokenization methods concurrently, using a lexicon to identify words and construct indices or classifiers, with the option to omit n-grams corresponding to identified words, implemented in computer hardware or software.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If lexical analysis tokenization is used, then word identification accuracy is improved, but completeness deteriorates due to incomplete lexicons
Solution Approach 1:
The patent combines lexical analysis tokenization and n-gram tokenization into a hybrid system. The lexical analyzer uses a lexicon to identify words accurately, while the n-gram tokenizer processes segments not found in the lexicon. The results from both methods are merged to produce a complete set of tokens, resolving the contradiction between accuracy and completeness.
2Reliability
If n-gram tokenization is used, then completeness is improved, but accuracy deteriorates due to spurious results
Solution Approach 1:
The patent introduces an intermediary filtering mechanism where the lexical analyzer's results are used to validate and filter n-gram tokens. When the lexical analyzer successfully identifies a word, corresponding n-grams are omitted from the index, preventing spurious results. This intermediary step maintains completeness while improving accuracy.
3Measurement precision
If both N-gram and lexical analysis tokenization are performed, then accuracy and completeness are improved, but system complexity increases
Solution Approach 1:
The patent segments the text processing task into two distinct components: lexical analysis for known words and n-gram analysis for unknown segments. By dividing the processing workload and applying different methods to different segments, the system achieves high accuracy without requiring a completely redundant dual-system architecture, thus managing complexity effectively.
Data Source
AI summary
A hybrid n-gram/lexical analysis tokenization system including a lexicon and a hybrid tokenizer operative to perform both N-gram tokenization of a text and lexical analysis tokenization of a text using the lexicon, and to construct either of an index and a classifier from the results of both of the N-gram tokenization and the lexical analysis tokenization, where the hybrid tokenizer is implemented in at least one of computer hardware and computer software and is embodied within a computer-readable medium.