Distributed Database Record Bucketing for Parallel Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed database systems face computational bottlenecks when comparing large data sets due to centralized repository access, leading to inefficiencies in resource usage and performance.
Innovation Solution
Distributed database system architecture that groups records across nodes into buckets, allowing each node to perform comparison operations in parallel, using derived data and blocking algorithms to assign records to buckets and execute comparison algorithms independently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a centralized repository is used to store and retrieve candidate record comparison data, then data consistency is maintained, but system performance deteriorates due to access bottlenecks
Solution Approach 1:
The centralized repository is segmented into distributed storage across multiple nodes in the distributed database system. Each node stores portions of the candidate record comparison data locally, eliminating the single-point bottleneck while maintaining data consistency through distributed consensus protocols.
Solution Approach 2:
The system transitions from a single-dimensional centralized storage model to a multi-dimensional distributed storage architecture. Data is organized across multiple dimensions including node distribution, bucket grouping, and record partitioning, enabling parallel access paths and eliminating the linear access bottleneck of centralized systems.
2Measurement precision
If pair-wise comparison is performed on all possible record pairs in large scale databases, then comprehensive comparison results are achieved, but computational resource requirements increase substantially
Solution Approach 1:
Candidate records are preprocessed and analyzed before the main comparison operation. Blocking algorithms and bucket grouping are applied in advance to identify and group only those records that are likely to match, eliminating the need to perform pair-wise comparison on all possible record pairs and reducing computational resources required.
Solution Approach 2:
Instead of performing exhaustive pair-wise comparison on all possible record pairs, the system applies partial action by comparing only candidate records that pass preliminary filtering. The blocking algorithm generates a subset of candidate pairs that are then compared, achieving sufficient comparison completeness without the excessive computational cost of exhaustive comparison.
3Productivity
If records are grouped into buckets and distributed across multiple nodes, then parallel processing capability is enhanced, but system complexity increases
Solution Approach 1:
Each node in the distributed system is designed with universal functionality to perform multiple operations: storing records, grouping into buckets, executing blocking algorithms, and performing pair-wise comparisons. This multi-functionality reduces the need for specialized components and simplifies the overall system architecture despite the distributed nature.
Solution Approach 2:
The system dynamically adjusts parameters such as bucket size, number of partitions, and distribution strategies based on data characteristics and system load. These parameter changes enable the system to optimize parallel processing capability while managing complexity through adaptive configuration rather than fixed complex structures.
Data Source
AI summary
Provided are a computer program product, system, and method for grouping records in buckets distributed across nodes a distributed database system to perform comparison of the grouped records. Upon receiving a record, data in the received record is processed to determine at least one containing bucket having attributes matching those of the received record, wherein the at least one containing bucket comprises at least one of a plurality of buckets, and wherein the buckets are assigned to the local node and the external nodes. A determination is made of at least one of the containing buckets assigned to at least one of the external nodes. At least a portion data in the received record is forwarded to each of the determined at least one external node to perform comparison matching with other records in the containing bucket at the external node.