Data Subset Clustering via Char Position Vectors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database management systems face challenges in scalable data-pattern matching and data-content matching, often resulting in false negatives and false positives due to reliance on schema matching and column name similarity, which are inefficient and inaccurate, especially when dealing with large collections of data sources.
Innovation Solution
The system generates char_position vectors for data-subsets, calculates proxy hash values, and performs data-pattern matching by assigning similar proxy hash values to clusters, using bitwise comparisons and vector similarity tests to group similar data-subsets, thereby improving accuracy and efficiency while reducing the search space for similar data-subsets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If pairwise comparisons of all data-subsets are performed to achieve accurate clustering, then clustering accuracy is improved, but system performance deteriorates due to computational bottleneck
Solution Approach 1:
The patent creates a simplified representation (copy) of each data-subset in the form of a vector that captures its essential characteristics. These vectors are then used for comparison instead of the original data-subsets, enabling efficient similarity computation while maintaining clustering accuracy. The vector representation acts as a proxy that preserves the necessary information for accurate grouping without requiring direct comparison of the full data-subsets.
Solution Approach 2:
The patent transforms the data-subsets into vector format, changing the representation parameter from raw data values to normalized vector representations. This parameter transformation enables the use of efficient mathematical operations (such as cosine similarity) to compute relationships between data-subsets, significantly improving computational performance while maintaining clustering precision.
2Ease of manufacture
If schema matching and column name similarity are used for data-pattern matching, then implementation simplicity is improved, but matching accuracy deteriorates due to false negatives and false positives
Solution Approach 1:
The patent replaces the mechanical approach of schema matching and column name comparison with a mathematical approach using vector representations and similarity calculations. Instead of relying on textual matching algorithms that are prone to false positives and negatives, the system uses vector-based mathematical operations to compute similarity, providing both accuracy and scalability.
Solution Approach 2:
The patent creates a universal vector representation method that can be applied to any data-subset regardless of its original format or content. This universal approach allows the same clustering mechanism to handle diverse data types uniformly, improving both implementation simplicity and matching accuracy by eliminating the need for type-specific matching logic.
3Productivity
If data-subsets are clustered based on pattern similarity to enable scalable analysis, then system scalability is improved, but clustering precision may deteriorate without proper similarity measurement
Solution Approach 1:
The patent creates vector copies of data-subsets that preserve their essential pattern characteristics. These vector copies enable efficient similarity computation through mathematical operations while maintaining the precision needed for accurate clustering. The vector representation captures the pattern information necessary for precise matching at scale.
Solution Approach 2:
The patent transforms data-subsets into vector format, changing the representation parameter to enable both scalability and precision. This parameter change allows the system to handle large numbers of data-subsets efficiently while maintaining accurate similarity measurement through standardized mathematical operations like cosine similarity.
Data Source
AI summary
The presently disclosed subject matter includes a computerized method and system of clustering data-subsets of one or more structured or semi-structured data sources, each data-subset comprising data-values. For each data-subset (e.g. column) in a group of data-subsets extracted from the one or more structured or semi-structured data sources, a respective vector indicative of characters' position distribution in data-values in the data-subset is generated, thereby giving rise to a group of vectors; for each vector in the group of vectors, a respective proxy hash value is calculated, and data-subsets of respective proxy hash values are assigned to clusters according to similarity between the respective proxy hash values and between the respective vectors.