N-gram Extraction Using Context Entropy and Confidence Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for n-gram identification and extraction often lose contextual information and sequence meaning, and fail to effectively handle semantically similar and related words.
Innovation Solution
A method and device that compute confidence scores and direction context entropy coefficients to identify and expand n-grams iteratively, using association and divergence measures to extract semantically linked n-grams, ensuring accurate preservation of context and meaning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If conventional methods are used for n-gram identification and extraction, then the process is simple and fast, but contextual information and sequence meaning are lost
Solution Approach 1:
The patent segments the n-gram extraction process into multiple stages: initial identification of candidate n-grams, iterative expansion in predefined directions (left/right contexts), confidence score computation at each stage, and semantic grouping. This segmented approach maintains contextual information by progressively expanding n-grams while evaluating their confidence scores, rather than extracting all n-grams at once which would lose context.
Solution Approach 2:
The patent performs preliminary actions by first identifying candidate n-grams and computing their initial confidence scores before expansion. The direction context entropy coefficients are calculated in advance to guide the iterative expansion process, ensuring that only high-confidence expansions are pursued, thereby preserving contextual relevance.
2Adaptability or versatility
If conventional methods extract n-grams, then extraction speed is maintained, but semantically similar and related words are not handled effectively
Solution Approach 1:
The patent implements feedback mechanisms by computing confidence scores for each n-gram and its expansions, using direction context entropy coefficients to evaluate semantic relevance. This feedback loop allows the system to iteratively refine n-gram selections, handling semantically similar words by comparing their confidence scores and retaining only those that maintain contextual meaning, thus improving semantic handling while maintaining reasonable extraction speed.
3Loss of information
If iterative expansion is performed to maintain context, then contextual information is preserved, but computational complexity increases
Solution Approach 1:
The patent applies partial action by performing iterative expansion only in predefined directions (left and right contexts) and only for n-grams that meet minimum confidence thresholds. Not all possible expansions are pursued; instead, the system selectively expands only those n-grams that are likely to maintain contextual relevance, thereby preserving context while limiting computational complexity.
Solution Approach 2:
The system performs preliminary computation of direction context entropy coefficients and initial confidence scores before iterative expansion. This preliminary action allows the system to pre-identify promising expansion directions and prune unlikely paths early, reducing the overall computational burden of iterative expansion while maintaining contextual preservation.
Data Source
AI summary
A method and device for n-gram identification and extraction is disclosed. The method includes identifying at least one n-gram from a sentence inputted by a user based on a confidence score associated with each of the at least one n-gram. The method further includes determining a direction context entropy coefficient for each of the at least one n-gram. The method includes iteratively expanding one or more of the at least one n-gram by the smallest n-gram unit at each iteration in a predefined direction in the sentence to generate at least one expanded n-gram, based on an associated direction context entropy coefficient. The method further includes extracting at each expanding iteration one or more of the at least one expanded n-gram based on an associated confidence score. The method includes grouping semantically linked n-grams from the one or more of the at least one expanded n-gram.