Multi-Cluster Duplicate Record Detection with Local Index Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale and multi-cluster data storage systems face challenges in performing cross-cluster queries and identifying duplicate or associated records, which require substantial compute resources, memory, and network bandwidth, leading to prolonged execution times and performance bottlenecks.
Innovation Solution
A duplicate record detector that initially queries each cluster separately, uses a multi-cluster index to reduce record sets, and applies matching logic to identify duplicates, reducing computational and memory resources by pre-filtering and leveraging a cross-cluster index to generate reduced record sets before applying complex matching logic.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If brute force searches are used to detect duplicate records in large-scale systems, then comprehensive duplicate detection is achieved, but execution time increases to hours or days and computational resources are excessively consumed
Solution Approach 1:
The patent segments the large-scale data storage system into multiple clusters, each handled independently. The duplicate detection process is divided into: (1) intra-cluster duplicate detection within each cluster separately, and (2) inter-cluster duplicate detection by comparing reduced record sets across clusters. This segmentation allows parallel processing and avoids the need for a single brute-force search across all data, dramatically reducing execution time while maintaining detection completeness.
Solution Approach 2:
The patent applies preliminary filtering using cluster-specific indexes before performing cross-cluster comparisons. Each cluster generates a reduced record set containing only potential duplicates based on local index matching. This preliminary action eliminates the need to compare every record across every cluster, reducing the search space from O(n²) to O(n log n) or better, thereby resolving the time-completeness contradiction.
2Measurement precision
If cross-cluster queries are performed to identify duplicate records distributed across multiple clusters, then comprehensive duplicate detection is achieved, but network bandwidth consumption increases and performance bottlenecks occur
Solution Approach 1:
The patent extracts only the necessary information for cross-cluster comparison by generating reduced record sets that contain solely the potential duplicates identified through local index matching. Instead of transferring entire clusters or all records across the network, only these minimized subsets are exchanged between clusters. This extraction principle reduces network bandwidth consumption from transmitting petabytes of data to transmitting merely the differential changes or match candidates, resolving the accuracy-bandwidth contradiction.
3Measurement precision
If all records from multiple clusters are retrieved and analyzed simultaneously, then complete duplicate detection is achieved, but computational resources and memory requirements become prohibitive
Solution Approach 1:
The patent segments the computational workload by handling each cluster independently first through local index-based filtering, then processing only the reduced cross-cluster candidates. This divides the monolithic O(n²) comparison problem into multiple smaller O(ni log ni) problems that can be solved with available resources, maintaining completeness while reducing device complexity requirements.
Solution Approach 2:
The patent performs preliminary filtering within each cluster using local indexes before cross-cluster analysis. This preliminary action identifies and eliminates records that cannot possibly be duplicates, reducing the input size for the computationally intensive cross-cluster matching phase. The result is a dramatic reduction in memory requirements and computational complexity while preserving the ability to detect all duplicates.
Data Source
AI summary
A duplicate record detector may retrieve and identify sets of corresponding records within a multi-cluster data storage system. The duplicate record detector initially may query each cluster to retrieve record sets including potentially duplicate records. The duplicate record detector then may use a multi-cluster index to reduce each of the initial record sets by determining which records have a corresponding potential duplicate record stored in another cluster. Matching logic may be used to compare and analyze the reduced record sets from each cluster, to determine duplicate records in other clusters using various matching criteria and including duplicate records having non-identical fields. The results of the duplicate record detector may be provided as output via a duplicate record report and/or to initiate automatic removal the duplicate records from one or more of the storage clusters.