Cross-Cluster Duplicate Record Detection Using Shared Index Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale and multi-cluster data storage systems face challenges in efficiently identifying and managing duplicate records due to significant compute resources, memory requirements, and network bandwidth issues, leading to prolonged execution times and performance bottlenecks.
Innovation Solution
A multi-step approach involving pre-filtering queries, a cross-cluster index, and matching logic is employed to retrieve and identify duplicate records, reducing initial record sets using a shared multi-cluster index and applying automated tools to determine duplicates, thereby optimizing computational and memory resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If brute force searches are used to detect duplicate records in large-scale systems, then comprehensive duplicate detection is achieved, but execution time increases to hours or days and compute resources are substantially consumed
Solution Approach 1:
The patent segments the large-scale data storage system into multiple clusters, each handled independently. The duplicate detection process is divided into: (1) local duplicate detection within each cluster, (2) extraction of candidate duplicate keys from local results, (3) coordination phase to resolve cross-cluster duplicates. This segmentation enables parallel processing and reduces the time complexity from O(n²) to approximately O(n), resolving the contradiction between comprehensive detection and execution time.
Solution Approach 2:
The patent performs preliminary actions by first conducting local duplicate detection within each cluster before attempting cross-cluster matching. Candidate duplicate keys are extracted and prepared in advance, and indexes are pre-built on these candidates. This preliminary processing reduces the data volume for subsequent cross-cluster coordination, significantly reducing overall execution time while maintaining detection completeness.
2Reliability
If cross-cluster queries are performed to analyze distributed data, then comprehensive data analysis is achieved, but network bandwidth consumption increases and performance is impacted by network delays
Solution Approach 1:
The patent extracts only the necessary candidate duplicate keys from local cluster results rather than querying all data across clusters. By extracting minimal necessary information (candidate keys and their cluster locations) and performing coordination only on these extracted elements, the network bandwidth consumption is dramatically reduced while maintaining the ability to detect all cross-cluster duplicates.
Solution Approach 2:
The patent segments the duplicate detection process into local and cross-cluster phases, minimizing cross-cluster communication to only the coordination of candidate keys. This segmentation reduces network bandwidth consumption from transmitting entire datasets to transmitting only candidate key information, resolving the contradiction between analysis completeness and network resource usage.
3Reliability
If duplicate detection is performed across multiple clusters distributed across different datacenters, then comprehensive duplicate detection is achieved, but the system complexity and coordination overhead increase
Solution Approach 1:
The patent segments the multi-cluster system into independent local detection units that each operate autonomously on their own data. Each cluster performs local duplicate detection independently, producing local results that are then coordinated through a standardized interface. This segmentation reduces system complexity by eliminating the need for complex distributed coordination protocols while maintaining cross-cluster detection capability.
Solution Approach 2:
The patent introduces an intermediary coordination phase that receives candidate duplicate keys from local clusters and performs centralized resolution. This intermediary layer simplifies the interaction between clusters by providing a standardized coordination mechanism, reducing the complexity of direct peer-to-peer cluster communication while ensuring comprehensive cross-cluster duplicate detection.
Data Source
AI summary
A duplicate record detector may retrieve and identify sets of corresponding records within a multi-cluster data storage system. The duplicate record detector initially may query each cluster to retrieve record sets including potentially duplicate records. The duplicate record detector then may use a multi-cluster index to reduce each of the initial record sets by determining which records have a corresponding potential duplicate record stored in another cluster. Matching logic may be used to compare and analyze the reduced record sets from each cluster, to determine duplicate records in other clusters using various matching criteria and including duplicate records having non-identical fields. The results of the duplicate record detector may be provided as output via a duplicate record report and/or to initiate automatic removal the duplicate records from one or more of the storage clusters.