Affix Discovery Using Patricia Trees for NLP
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems for automatically identifying affixes in words are inefficient due to computational complexity, limited ability to discover both prefixes and suffixes simultaneously, inability to find nested affixes, and failure to recognize affixes with non-alphabetic characters, which hinders effective morphological analysis and out-of-vocabulary handling in natural language processing.
Innovation Solution
A computer system and method using Patricia trees to represent words, integrating prefix and suffix discovery, allowing for the identification of affixes without precompiled lists, and capable of finding long and nested affixes, including those with non-alphabetic characters, by iteratively refining potential affixes and utilizing knowledge about prefixes and suffixes to enhance discovery.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional split-all method is used to parse words into prefix and stem or stem and suffix, then all possible splits are considered, but computational complexity becomes excessive and processing time increases significantly
Solution Approach 1:
The patent segments the word parsing problem into two independent parts: prefix discovery and suffix discovery. Instead of considering all possible splits simultaneously (split-all method), the system separately identifies prefixes by analyzing word beginnings and suffixes by analyzing word endings. This segmentation reduces computational complexity from O(n*m) to O(n) + O(m), significantly improving processing speed while maintaining completeness of affix discovery.
Solution Approach 2:
The patent extracts and removes affixes from words in a systematic manner. Once prefixes are identified and removed, the remaining stem is used for suffix analysis. This extraction approach avoids redundant computations by eliminating already-processed parts from further analysis, thereby improving processing efficiency without sacrificing measurement precision.
2Reliability
If precompiled lists of affixes and morphological rules are used, then morphological analysis can be performed, but the system cannot identify newly created words and requires manual compilation which is time-intensive
Solution Approach 1:
The patent implements a self-service mechanism where the system automatically discovers and builds its own affix lists from the input text without requiring precompiled dictionaries. By analyzing patterns in the text data itself, the system identifies prefixes and suffixes that are actually used, making it both reliable for known words and adaptable to new words. This eliminates the need for manual compilation while maintaining analysis accuracy.
Solution Approach 2:
The patent performs preliminary action by pre-processing the text to identify and extract potential affixes before main morphological analysis. This preliminary discovery phase builds a customized affix list specific to the domain text, enabling the system to handle both standard and newly created words effectively without requiring external precompiled lists.
3Extent of automation
If simple genetic algorithm with MDL as fitness function is used for suffix finding, then automatic identification is achieved, but the algorithm is limited to hundreds of words and takes eight and a half hours to process 120 words
Solution Approach 1:
The patent replaces the genetic algorithm mechanism with a direct pattern-matching approach. Instead of using evolutionary computation with fitness functions and generations, the system directly analyzes word patterns to identify affixes. This substitution of mechanical system eliminates the computational overhead of genetic algorithms while maintaining automatic identification capability, dramatically improving processing throughput from hours to much faster times.
Solution Approach 2:
The patent changes the fundamental parameters of the search approach by transitioning from probabilistic evolutionary search to deterministic pattern analysis. This parameter change allows the system to handle large numbers of words efficiently by using direct linguistic pattern recognition rather than iterative optimization, thereby increasing productivity while preserving automation.
4Productivity
If affix length is limited to reduce search space size, then computational complexity is reduced, but longer affixes found in technical documents cannot be discovered
Solution Approach 1:
The patent implements dynamic affix length adjustment where the maximum affix length is not fixed but adapts based on the actual data. The system determines appropriate affix lengths dynamically from the input text, allowing it to discover both short and long affixes as needed. This dynamic approach maintains computational efficiency by only analyzing lengths that actually appear in the data, while preserving the ability to find long technical affixes when present.
Data Source
AI summary
A computer system and a method for analyzing text in one or more electronic documents are disclosed. The computer system comprises one or more system interfaces; and an affix process that determines one or more affixes of one or more words in one or more of the documents and provides the affixes to the system interface. The preferred embodiment of the invention may be used to build a domain specific morphology lexicon for NLP applications so that they can recognize out-of-vocabulary words. The disclosed procedure utilizes the fact that the processes of discovering prefixes and suffixes are not independent. Many words, especially in technical documents, have complex morphological structures, and thus the knowledge about prefixes helps the discovery of suffixes and vice versa.