Clustering Category Data Using Entropic Similarity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing clustering and classification algorithms face challenges in handling category datasets with discrete attributes lacking a natural similarity measure, making it difficult to effectively organize and index multimedia data such as recorded TV programs and electronic program guide entries.
Innovation Solution
A method and apparatus that cluster and classify category data by representing it as a hierarchy of clusters and using a folder-based classification system, employing semantic augmentation, agglomerative hierarchical clustering, and entropic similarity conditions to generate a taxonomy and classify data efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional clustering algorithms are used on category datasets, then the algorithms work well when data is numerical with inherent similarity metrics, but they fail to effectively handle category data with discrete attributes lacking natural distance measures
Solution Approach 1:
The patent transforms category data into a numerical probability distribution space by computing the probability of each attribute value occurring within a cluster. This parameter transformation enables the application of numerical clustering algorithms to category data by changing the representation from discrete categories to continuous probability values, thereby resolving the incompatibility between traditional algorithms and category data types.
2Loss of information
If category data is organized without semantic augmentation, then the data structure remains simple, but the information content is insufficient leading to data sparsity
Solution Approach 1:
The patent performs preliminary semantic augmentation by pre-computing attribute probability distributions for each category and storing them in a lookup table. This preliminary action enriches the data with semantic information before clustering operations, reducing data sparsity and improving information content without adding complexity during the actual clustering process.
3Ease of operation
If hierarchical clustering is applied to category data, then data organization improves through cluster hierarchies, but the computational complexity increases due to the need to compute and compare probability distributions
Solution Approach 1:
The patent pre-computes and stores attribute probability distributions for all categories in a lookup table before the hierarchical clustering process. This preliminary computation avoids redundant calculations during cluster merging operations, reducing the computational complexity of hierarchical clustering while maintaining the benefits of hierarchical data organization.
Solution Approach 2:
The patent uses probability distribution copies from the lookup table during clustering operations instead of re-computing distributions from raw data. This copying approach significantly reduces computational complexity by avoiding repeated calculations of attribute probabilities while maintaining the accuracy of hierarchical cluster comparisons.
Data Source
AI summary
Records including category data is clustered by representing the data as a plurality of clusters, and generating a hierarchy of clusters based on the clusters. Records including category data are classified into folders according to a predetermined entropic similarity condition.