Topic Evaluation Metrics for Automated Text Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data management systems fail to effectively capture contextual relationships between words, leading to unreliable and time-consuming human interpretation for determining the optimal number of topics in text data, as existing evaluation metrics like coherence and perplexity scores are insufficient for assessing topic quality and often yield non-interpretable results.
Innovation Solution
A processor-implemented method and system that calculates a topic score by determining a word score based on probability, a litany index value for repetitive tokens, and a contiguity score to identify closely connected topics, allowing for automated validation and improved reliability in determining contextually associated words and topics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If coherence measures are used to evaluate topic quality, then semantic interpretability is improved, but the metrics are limited to specific APIs and algorithms reducing versatility
Solution Approach 1:
The patent creates evaluation metrics that work across multiple APIs and algorithms by defining universal concepts like topic coherence, topic diversity, and topic coverage that can be calculated regardless of the specific topic modeling approach used. This allows the same evaluation framework to assess topics from different APIs (e.g., gensim, sklearn) and algorithms (e.g., LDA, NMF, SVD) uniformly.
Solution Approach 2:
The patent segments the evaluation process into distinct measurable components: topic coherence (semantic similarity), topic diversity (word distribution), and topic coverage (representation quality). Each component can be independently calculated and combined, allowing flexibility in applying different calculation methods while maintaining overall evaluation consistency across various APIs.
2Measurement precision
If perplexity score is optimized to improve predictive likelihood, then model prediction accuracy is improved, but human interpretability of topics deteriorates
Solution Approach 1:
The patent introduces feedback mechanisms where evaluation metrics (coherence, diversity, coverage) are calculated and used to adjust topic modeling parameters. This feedback loop allows optimization of both predictive likelihood and human interpretability by iteratively refining topics based on multiple evaluation criteria rather than relying solely on perplexity.
Solution Approach 2:
The patent changes evaluation parameters from solely relying on perplexity to using a composite of multiple metrics including topic coherence, topic diversity, and topic coverage. This parameter change enables balancing between predictive accuracy and human interpretability by weighting different metrics according to their importance for the specific application.
3Measurement precision
If multiple evaluation metrics are calculated to improve topic assessment accuracy, then measurement precision is improved, but computational complexity increases
Solution Approach 1:
The patent implements partial evaluation by allowing users to select only the metrics and components needed for their specific application. Rather than calculating all possible metrics uniformly, the system enables partial calculation of coherence, diversity, and coverage metrics based on available data and user requirements, reducing unnecessary computational overhead while maintaining assessment accuracy.
4Reliability
If topic coherence is measured by semantic similarity to improve interpretability, then topic quality is improved, but the metric cannot be applied across different APIs reducing versatility
Solution Approach 1:
The patent defines topic coherence in a universal manner that can be calculated across different APIs by using standardized word embedding approaches and similarity calculations. The coherence metric computes semantic similarity between words in topics using methods that are API-agnostic, allowing the same coherence calculation to be applied to topics from gensim, sklearn, and other APIs without modification.
Data Source
AI summary
Rapid development in digitization results in generation of huge amounts of unstructured text data by most organizations from several sources. A major drawback of using existing metrics in the data management is assessing quality of topics associated with context in a document. A processed dataset obtained from a raw dataset is mapped to obtain a relationship between one or more words from one or more topics. A word score is determined based on probability of number of constructs in sentences. At least one repetitive word from the one or more topics are determined. A topic score with degree of contextual association between the one or more words in each topic is determined by mean value of the word score. A contiguity score with degree of contextual association between the one or more topics is determined based on median value of the topic score to obtain one or more connected topics.