Reference Database Construction for Tractable Data Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data classification techniques face computational and storage resource limitations when dealing with large datasets, leading to prolonged processing times and memory constraints, and random or statistical subset selection can result in inaccurate representations of the complete data set.
Innovation Solution
A method to construct a reference database by iteratively selecting data items with similarities less than a target difference threshold, reducing the dataset while maintaining classification accuracy, which involves determining differences between data items and moving similar items to a reference database, thereby reducing the computational and storage burdens.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a complete set of data items is used for classification, then classification accuracy is improved, but computational time and storage requirements increase
Solution Approach 1:
The patent extracts only the necessary subset of data items from the complete set to form a reference database. By selectively choosing representative data items rather than using the entire dataset, the system achieves accurate classification while reducing computational time and storage requirements.
Solution Approach 2:
The patent applies partial action by using a subset of data items rather than the complete set. The iterative selection process identifies the minimum necessary portion of data required to maintain classification accuracy, avoiding the excessive computational burden of processing all available data.
2Measurement precision
If a complete set of data items is stored in memory, then classification accuracy is improved, but memory constraints are exceeded
Solution Approach 1:
The patent extracts a representative subset of data items from the complete set to create a reference database that fits within memory constraints. This selective extraction maintains classification accuracy while reducing the quantity of data stored in memory to manageable levels.
Solution Approach 2:
The system uses partial action by storing only the necessary subset of data items in memory rather than the complete set. The iterative selection process ensures that the reference database contains sufficient data for accurate classification without exceeding available memory capacity.
3Productivity
If random or statistical subset selection is used, then processing time is reduced, but representation accuracy of the complete dataset deteriorates
Solution Approach 1:
The patent employs feedback through an iterative selection process that evaluates the quality of the reference database. By continuously assessing whether selected data items accurately represent the complete set and adjusting selections based on this feedback, the system achieves both fast processing and high representation accuracy.
Solution Approach 2:
The system uses dynamics by implementing an adaptive iterative selection process rather than static random sampling. The selection criteria and thresholds are dynamically adjusted to ensure the reference database maintains accurate representation of the complete dataset while enabling efficient processing.
Data Source
AI summary
For each unique pair of a complete set of data items, a computing device determines a distance between the data items of the unique pair. The computing device repeats the following until no data items remain in the complete set. For each data item remaining in the complete set, the computing device determines a similarity subset including each other data item that the distance between the data item and the other data item is less than a target difference threshold. The computing device moves a selected data item from a largest similarity subset to a reference database that is a subset of the complete set. The computing device removes each data item from the complete set that the distance between the selected data item and the data item is less than the threshold. A new data item can be classified using the reference database.