Fingerprint-Based Data Classification for Large Datasets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing automated data classification methods require significant computing resources and time, especially when dealing with large data sets, as they analyze each column against every data class, leading to high computing costs and inefficiencies.
Innovation Solution
The method generates a fingerprint for each data column based on predetermined rules, comparing it to target fingerprints associated with classes, and uses fingerprint trees to define priority rules, allowing for efficient classification by comparing only a subset of class dimensions, thereby reducing computational load.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If full data classification methods analyze every row of data in a data column against every elected class, then classification accuracy is improved, but computing resources and time are significantly increased
Solution Approach 1:
The patent segments the classification process into two distinct phases: (1) fingerprint generation phase where a condensed representation (fingerprint) is created for each data column and target class, and (2) fingerprint matching phase where only these compact fingerprints are compared. This segmentation transforms the original O(M×N×R) complexity into O(M×N+F×M) complexity, where F is the small number of fingerprints per class, thereby maintaining accuracy while dramatically improving classification speed for large datasets.
Solution Approach 2:
The patent creates simplified copies (fingerprints) of the original data columns and target classes that capture essential classification characteristics without containing the full data. These fingerprints serve as proxies that can be quickly compared and matched, eliminating the need to process the complete dataset during classification while preserving the ability to accurately determine data class membership.
2Reliability
If full data classification processes are performed on large data sets, then comprehensive classification coverage is achieved, but computing costs and time requirements increase significantly
Solution Approach 1:
The patent performs preliminary action by pre-generating fingerprints for all target classes and storing them in a fingerprint repository before the actual classification task. This preliminary preparation allows the classification process to proceed by simply comparing data column fingerprints against the pre-computed target class fingerprints, eliminating the need to re-analyze the entire dataset for each classification operation and significantly reducing classification time for large datasets.
3Measurement precision
If conventional full classification methods are used, then all data classes are thoroughly evaluated, but computational efficiency deteriorates
Solution Approach 1:
The patent extracts the essential classification characteristics from the full datasets and encapsulates them in compact fingerprint representations. By taking out only the critical features needed for classification (stored in fingerprints) and removing unnecessary data, the system maintains thorough classification evaluation while dramatically reducing computational resource consumption during the matching phase.
Data Source
AI summary
Systems and methods are provided for automated classification of data using fingerprints. In embodiments, a method includes: generating, by a computing device based on predetermined rules, a fingerprint of a data column in a data set to be classified, the fingerprint comprising dimensions, wherein each of the dimension is assigned an attribute representing a characteristic of data in the data column; determining, by the computing device, that the fingerprint matches one or more target fingerprints by comparing the fingerprint to the target fingerprints, wherein each target fingerprint is associated with a class and includes dimensions, and each dimension is assigned an attribute representing a characteristic of data in the class; and assigning, by the computing device, one or more classes to the data column based on the one or more target fingerprints, thereby generating classified data.