Entropy-Based Term Dominance Metric for Text Corpus Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current information retrieval systems struggle to understand individual user needs and detect surprising information, as they rely on keyword queries and statistical composition of corpora, failing to account for the complex factors influencing a person's expertise and motivations.
Innovation Solution
An entropy-based term dominance metric is developed to characterize text corpora, allowing for the comparison of term dominance across different document sets and modeling user expertise through statistical properties of text, including term frequency, distribution, context, and associations, using the STANLEY library to analyze eye movements and physiological reactions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If keyword-based search engines are used to retrieve documents, then the system can efficiently retrieve documents containing queried terms, but the system cannot understand user information needs or detect surprising information
Solution Approach 1:
The patent transforms the retrieval approach by changing from keyword matching to entropy-based term dominance analysis. It calculates dominance values for terms in retrieved documents compared to background corpora, using entropy measurements to identify terms that are unexpectedly dominant. This parameter change enables the system to detect surprising information while maintaining retrieval efficiency.
2Productivity
If statistical composition of corpora is used to determine relevance, then documents can be ranked by term frequency, but the system cannot account for individual user expertise and motivations
Solution Approach 1:
The patent applies local quality by calculating term dominance separately for different user domains or expertise areas. It maintains background corpora specific to different domains and calculates entropy-based dominance values relative to domain-specific expectations. This allows the system to adapt relevance determination to individual user expertise while maintaining efficient statistical processing.
3Measurement precision
If entropy-based term dominance metric is calculated across corpora of different sizes, then term dominance can be compared between document sets, but the metric must be normalized to account for corpus size differences
Solution Approach 1:
The patent achieves equipotentiality by normalizing term dominance values to a common scale that is independent of corpus size. It uses entropy calculations that inherently account for probability distributions, allowing fair comparison of term dominance across corpora of different sizes. This normalization approach equalizes the measurement potential across different corpus scales without excessive computational complexity.
Data Source
AI summary
An improved entropy-based term dominance metric useful for characterizing a corpus of text documents, and is useful for comparing the term dominance metrics of a first corpus of documents to a second corpus having a different number of documents.