N-gram Tree Algorithm for Multi-word Term Frequency Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional document processing algorithms are inflexible in determining the frequencies of multi-word phrases of varying lengths and are constrained by grammatical and syntactical models, limiting their ability to identify relevant keywords and phrases effectively.
Innovation Solution
The use of an n-gram tree algorithm and data structure allows for the efficient identification of multi-term phrases of varying lengths by determining their frequencies and positional data within a document, enabling the selection of more relevant keywords and phrases without pre-defined parameters, and filtering out non-relevant phrases.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional document processing algorithms are used to determine keyword frequencies, then the processing can be performed with simple frequency counting, but the system is inflexible when it comes to determining frequencies of multi-word phrases of varying lengths
Solution Approach 1:
The patent segments the document processing task into hierarchical levels based on phrase length (unigrams, bigrams, trigrams, etc.). Each level processes phrases of a specific length, allowing the system to handle multi-word phrases of varying lengths systematically. This segmentation enables flexible frequency determination for phrases of any length without requiring a completely different processing approach for each phrase length.
Solution Approach 2:
The patent implements dynamic processing where the system can adaptively determine frequencies for phrases of different lengths (n-grams) based on the document content. The processing dynamically adjusts to handle variable phrase lengths by implementing iterative frequency counting across multiple passes, where each pass targets phrases of a specific length range, enabling the system to be versatile without requiring all parameters to be pre-specified.
2Adaptability or versatility
If conventional POS tagger algorithms are used for keyword extraction, then grammatical and syntactical constraints can be applied, but the system cannot determine frequencies of multi-word phrases without pre-specified parameters
Solution Approach 1:
The patent performs preliminary frequency counting for all possible phrases up to a maximum length n before applying any filtering or ranking. This preliminary action collects frequency data for multi-word phrases of varying lengths without requiring pre-specified parameters about which phrases to search for. The system prepares comprehensive frequency information in advance, enabling flexible query responses later without re-processing the document.
Solution Approach 2:
The patent changes the parameter of phrase length consideration from fixed to variable by implementing iterative frequency determination for different n-gram levels. Instead of being constrained to a single phrase length parameter, the system processes and determines frequencies for phrases of lengths 1 through n, allowing operational flexibility to query frequencies for any phrase length without reconfiguration.
3Measurement precision
If comprehensive frequency analysis of all possible phrases is performed, then relevant keywords can be identified, but the computational resources and processing time increase significantly
Solution Approach 1:
The patent implements partial action by determining frequencies for phrases up to a practical maximum length n rather than analyzing every possible phrase combination in the document. This partial processing approach achieves sufficient measurement precision for keyword identification while avoiding the computational burden of exhaustive analysis. The system performs frequency counting for n-grams where n is a reasonable limit, providing accurate results without excessive processing time.
Solution Approach 2:
The patent segments the frequency analysis into iterative passes, where each pass handles phrases of a specific length. This segmentation allows the system to process frequencies for unigrams, then bigrams, then trigrams, and so on, rather than attempting to analyze all phrases simultaneously. This approach improves measurement precision for multi-word phrases while managing computational resources by breaking down the exhaustive analysis into manageable segments.
Data Source
AI summary
A term analyzer receives an ordered collection of text-based terms. The term analyzer analyzes groupings of consecutive text-based terms in the ordered collection to identify occurrences of different combinations of text-based terms. In addition, the term analyzer maintains frequency information representing the occurrences of the different combinations of text-based terms in the collection. The frequency information can then be used to determine relatively significant keywords and/or keyword phrases in the document. In an example configuration, the term analyzer creates a tree in which a first term in a given grouping of the groupings is defined as a parent node in the tree and a second term in the given grouping is defined as a child node of the parent node in the tree. The method of the analyzer generalizes to create a tree of multi-word terms in which the terms can be efficiently ranked by occurrence.