Duplicate Dataset Identification Using Multi-Stage Distance Metrics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Organizations face challenges in identifying and managing duplicate datasets within large pools of heterogeneous datasets, which leads to increased maintenance and storage costs due to the variability in data types and formats.
Innovation Solution
A system that compares schema information and metadata to generate name-based similarity scores, and data distribution information to generate data distribution similarity scores, calculating an overall distance metric to identify clusters of similar datasets and illustrate inferred lineage, thereby enabling effective sorting and management of duplicate datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If organizations manually review and compare datasets to identify duplicates, then accuracy in identifying true duplicates can be maintained, but the time and computational resources required increase significantly with large pools of heterogeneous datasets
Solution Approach 1:
The patent segments the duplicate identification process into multiple hierarchical stages: first comparing metadata and schema information to generate initial similarity scores, then progressively comparing data distribution statistics, and finally performing detailed data-level comparisons only on candidate pairs that pass earlier filters. This segmentation allows the system to quickly eliminate non-duplicate pairs using lightweight comparisons while reserving computational resources for thorough analysis of promising candidates, thereby maintaining high accuracy while reducing overall processing time.
Solution Approach 2:
The patent performs preliminary comparisons using metadata, schema information, and data distribution statistics before conducting full data-level comparisons. By pre-computing similarity scores based on these preliminary features and filtering out pairs with scores below thresholds, the system prepares the dataset in advance to avoid unnecessary detailed comparisons, thus significantly reducing the time required for complete duplicate identification while preserving accuracy for true duplicates.
2Reliability
If organizations implement comprehensive duplicate detection across all datasets, then data quality and storage efficiency improve, but the computational complexity and resource requirements increase
Solution Approach 1:
The patent divides the comprehensive duplicate detection process into distinct computational stages with increasing depth of analysis. Each stage processes datasets using progressively more computationally intensive methods only on subsets of data that pass previous stages. This segmentation reduces overall computational complexity by avoiding full detailed comparisons across all dataset pairs while still achieving comprehensive coverage of true duplicates through the multi-stage filtering approach.
Solution Approach 2:
The patent applies partial action by performing complete detailed comparisons only on candidate pairs that meet similarity thresholds from preliminary comparisons, rather than exhaustively comparing all possible pairs. This selective approach achieves sufficient duplicate detection coverage (partial action) without the prohibitive computational cost of exhaustive comparison, thereby improving data quality while managing computational complexity.
3Productivity
If organizations store all datasets without deduplication, then data availability and access speed are maintained, but storage costs and maintenance overhead increase
Solution Approach 1:
The patent performs preliminary duplicate identification using metadata and data distribution comparisons before finalizing deduplication decisions. By pre-identifying candidate duplicates and validating them through staged comparisons, the system can confidently remove confirmed duplicate copies while retaining at least one copy of each unique dataset, thus reducing storage space while maintaining data availability for all unique datasets.
Solution Approach 2:
The patent changes the state of datasets from redundant multiple copies to optimized single copies by identifying and removing duplicates. This parameter change in data representation (from multiple identical instances to single unique instances) directly reduces storage space requirements while maintaining data accessibility, as the system preserves one copy of each unique dataset that can serve all access needs.
Data Source
AI summary
An embodiment for identifying and sorting duplicate datasets within a large pool of heterogeneous datasets may include received a plurality of heterogeneous datasets. The embodiment may automatically compare schema information and metadata within each of the received plurality of heterogeneous datasets to generate name-based similarity scores for each dataset. The embodiment may also automatically compare data distribution information within each of the received plurality of heterogeneous datasets to generate a plurality of data distribution similarity scores for each heterogeneous dataset. The embodiment may further include automatically calculating an overall distance metric using the name-based similarity scores and plurality of data distribution similarity scores. The embodiment may also include based on the calculate overall distance metric, automatically generating distance graphs that identifying clusters of similar datasets and illustrate inferred lineage for the clusters of similar datasets.