Dynamic Data Classification via Term Frequency Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data classification methods are inefficient and often fail to classify data values due to the need for pre-defined classes and logic, leading to incomplete or inaccurate classification, especially when dealing with large datasets or unknown classes.
Innovation Solution
A computer-implemented method that determines candidate classes by identifying common terms across multiple knowledge bases, using a lookup query to find terms present in multiple datasets, and returning the terms with the smallest number of hits, allowing for dynamic classification without pre-defined classes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If pre-defined classes and logic are used for data classification, then classification accuracy can be maintained for known classes, but the system fails to classify unknown classes and requires complex pre-defined logic
Solution Approach 1:
The system performs self-classification by automatically determining classes through knowledge base queries and term frequency analysis, eliminating the need for pre-defined classification logic. The classifier queries knowledge bases for each data value, counts term occurrences, and automatically determines classes based on frequency thresholds, allowing the system to adapt to unknown classes without human intervention.
Solution Approach 2:
The system changes the classification approach from static pre-defined classes to dynamic class determination based on term frequency parameters. By querying multiple knowledge bases and analyzing term occurrence frequencies, the system adapts its classification criteria based on actual data characteristics, enabling classification of unknown classes through parameter-driven decision making rather than fixed rules.
2Productivity
If traditional data classification methods are used, then known classes can be classified accurately, but the rate of successfully classified data values decreases for large datasets with unknown classes
Solution Approach 1:
The system continuously queries knowledge bases for each data value and continuously updates term frequency counts, maintaining an ongoing classification process rather than relying on pre-computed classes. This continuous action allows the system to process large datasets efficiently while adapting to unknown classes, improving both productivity and reliability through persistent analysis.
Solution Approach 2:
The system incorporates feedback mechanisms by analyzing term frequency results from knowledge base queries and adjusting classification decisions accordingly. The feedback loop compares term occurrence frequencies against thresholds and updates class assignments dynamically, ensuring that both known and unknown classes are accurately identified while maintaining high classification rates across large datasets.
3Measurement precision
If multiple knowledge bases are queried for each data value, then classification accuracy improves, but processing time increases
Solution Approach 1:
The system performs preliminary actions by pre-processing knowledge bases and pre-computing term frequencies before actual classification. By preparing knowledge base indices and term occurrence counts in advance, the system reduces the time required during classification operations, maintaining high accuracy while minimizing processing time through advance preparation.
Solution Approach 2:
The system creates a simplified representation (copy) of knowledge base information through term frequency counting and indexing. Instead of querying entire knowledge bases for each data value, the system uses pre-extracted term frequency data as a copy or summary, significantly reducing processing time while preserving the essential information needed for accurate classification.
Data Source
AI summary
The invention relates to a computer-implemented method for classifying a set of data values. For each of the data values of the set of data values, a set of one or more terms associated with the respective data value is determined using one or more first knowledge bases. A set of common terms is determined. The set of common terms comprises terms present in more than one of the sets of terms. For each of the common terms, a number of hits for a lookup query against one or more second knowledge data bases is determined. One or more common terms of the set of common terms with the smallest number of hits are determined and a result is returned. The result comprises the one or more common terms with the smallest number of hits as one or more candidate classes for classifying the set of data values.