Data Set Clustering via Column Attribute Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
It is computationally expensive and impractical to compare each file or unit of data across multiple data sets to determine their similarity, making it difficult to efficiently classify and treat data sets with similar attributes uniformly.
Innovation Solution
A system that analyzes column-specific attributes of data sets to generate column identifiers, clusters data sets based on common columns, and performs pairwise comparisons within clusters to determine related data sets, reducing the number of comparisons needed and ensuring similar data sets are treated consistently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If each file or unit of data of each data set is compared against the same unit of data of every other data set to determine similarity, then measurement precision is improved, but computational cost increases excessively
Solution Approach 1:
The patent segments data sets into multiple blocks and compares corresponding blocks between data sets rather than comparing entire data sets at once. This division into smaller units reduces the computational complexity from O(n²) to more manageable levels while maintaining accuracy through systematic block-by-block comparison and aggregation of similarity scores.
Solution Approach 2:
The patent performs preliminary actions by first identifying and comparing common columns or attributes between data sets before conducting full data comparison. This preliminary filtering step reduces the search space and identifies candidate related data sets that warrant more detailed comparison, thereby reducing overall computational cost while preserving measurement precision for relevant comparisons.
2Reliability
If comprehensive comparison of all data sets is performed to ensure accurate classification, then reliability is improved, but productivity decreases
Solution Approach 1:
The patent applies partial action by performing comparison operations on only the necessary portions of data sets. Instead of exhaustively comparing every element of every data set against every other data set, the system performs partial comparisons on representative blocks and uses threshold-based stopping criteria, achieving sufficient classification reliability without the full computational burden of complete comparisons.
Solution Approach 2:
The patent segments the classification process into multiple passes: first identifying common columns, then grouping data sets by shared attributes, and finally performing detailed comparisons only within groups. This segmented approach maintains classification reliability by ensuring thorough comparison where needed while improving productivity by avoiding unnecessary comparisons between unrelated data sets.
3Measurement precision
If data sets are clustered based on common columns with rigorous pairwise comparisons, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The patent segments the complex task of related data set identification into distinct modular components: a column analysis module that identifies common columns, a clustering module that groups data sets by shared attributes, and a pairwise comparison module that performs detailed similarity assessment. This segmentation reduces device complexity by organizing functionality into independent, manageable modules while maintaining measurement precision through systematic processing at each stage.
Solution Approach 2:
The patent introduces intermediary structures such as column identifier mappings and group membership data structures that mediate between raw data sets and final similarity determinations. These intermediaries simplify the overall system complexity by pre-processing and organizing data before final comparison, reducing the computational burden on the core comparison logic while improving identification accuracy.
Data Source
AI summary
Determination of related data sets is disclosed, including: analyze a first plurality of columns belonging to a first data set by determining first attributes for each column in the first data set; analyze a second plurality of columns belonging to a second data set by determining second attributes for each column in the second data set; determine a set of common columns belonging to the first data set and the second data set by comparing at least a portion of the first attributes for each column in the first data set to at least a portion of the second attributes for each column in the second data set; and cluster a plurality of data sets including the first data set and the second data set based at least in part on the set of common columns.