Text Classification Using Modified Jaccard Distance Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine-based processes for automatically classifying large numbers of text entries into thousands of categories are excessively time-consuming and computationally intensive, becoming unstable when the number of candidate classes exceeds one thousand.
Innovation Solution
A control circuit that uses a modified Jaccard distance to generate groups of predefined classes, calculates probabilities for text entry membership, and employs machine learning processes to quickly classify additional text entries into predefined classes, reducing the number of groups by one or two orders of magnitude and completing classification in significantly less time than prior art approaches.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If known machine-learning techniques are used to classify text entries into thousands of categories, then automatic classification capability is achieved, but the process becomes extremely time consumptive and computationally intensive
Solution Approach 1:
The patent segments the classification process into two distinct stages: (1) a clustering phase that groups similar text entries together using computational algorithms, and (2) a classification phase that assigns pre-defined class labels to the clustered groups. This segmentation reduces the overall classification time by handling the computationally intensive clustering separately from the label assignment process.
Solution Approach 2:
The patent performs preliminary clustering of text entries before applying class labels. By pre-grouping similar entries into clusters based on their content characteristics, the system reduces the search space for subsequent classification, thereby decreasing the time required for the overall classification process while maintaining automation.
2Extent of automation
If known machine-learning techniques are used to classify text entries into thousands of categories, then automatic classification capability is achieved, but computational resources are excessively consumed
Solution Approach 1:
The patent divides the classification task into clustering and classification phases, allowing computational resources to be optimized for each phase separately. The clustering phase uses efficient algorithms to group entries, reducing the overall computational burden compared to applying full machine-learning classification to every entry individually.
Solution Approach 2:
The patent extracts and applies pre-defined class labels to clustered groups rather than performing full classification on each individual entry. This extraction approach reduces computational resource consumption by leveraging the structure already created during clustering, avoiding redundant computations.
3Adaptability or versatility
If the number of candidate classes exceeds one thousand, then comprehensive classification coverage is achieved, but the machine-learning models become unstable
Solution Approach 1:
The patent segments the large number of classes into manageable clusters of similar text entries. By grouping entries into clusters first, the system reduces the effective number of categories that need to be processed simultaneously, thereby stabilizing the machine-learning models while still maintaining comprehensive classification coverage across all thousand+ classes.
Solution Approach 2:
The patent introduces a hierarchical dimension to the classification process by organizing classes into clusters. This additional organizational layer transforms the flat, high-dimensional classification problem into a structured hierarchy, making the models more stable when dealing with thousands of classes while preserving comprehensive coverage.
Data Source
AI summary
A control circuit obtains and utilizes a first set of rules that determines a distance, as a function of a modified Jaccard distance, between each of a number of predetermined classes. The control circuit utilizes that information to generate groups of the predefined classes by evaluating the distance between each of the predetermined classes. First probabilities for each of the additional text entries regarding membership in each of the groups so generated is calculated along with second probabilities regarding membership in a particular one of the predefined classes in its corresponding group. A resultant probability is then calculated using the first and second probabilities that a given additional text entry belongs to a particular one of the predefined classes. The control circuit then utilizes that resultant probability information to automatically classify additional text entries by associating the given additional text entry to a particular one of the predefined classes.