Database Record Clustering for De-duplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database systems face significant performance challenges when identifying and removing duplicate records, especially in large datasets, as existing methods are computationally intensive and inefficient, particularly in multi-tenant CRM systems with millions of records.
Innovation Solution
The method involves generating clusters of records based on keys, splitting them into subsets, identifying duplicate sets within these subsets, merging duplicate sets, and forming representative records to efficiently identify and remove duplicates, thereby improving processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If records are compared one-by-one to identify duplicates, then duplicate identification is thorough and accurate, but processing time and computational resources increase significantly
Solution Approach 1:
The patent divides the database records into multiple clusters based on common attributes or keys. Each cluster contains a subset of records that are grouped together, allowing duplicate identification to be performed on smaller portions of data rather than comparing all records against each other. This segmentation reduces the computational complexity from O(n²) to approximately O(n), significantly improving processing efficiency while maintaining duplicate identification accuracy.
2Reliability
If all records are processed to ensure complete duplicate removal, then database integrity is maximized, but computational resources and processing time are excessively consumed
Solution Approach 1:
Records are organized into clusters based on shared attributes, enabling the system to process only relevant subsets of records for duplicate identification. This approach maintains comprehensive duplicate removal within each cluster while avoiding unnecessary comparisons across unrelated records, thus preserving database integrity without consuming excessive computational resources.
Solution Approach 2:
The patent performs preliminary clustering of records based on common attributes before conducting duplicate identification. This preliminary organization groups potentially duplicate records together, so that the subsequent duplicate detection process only needs to examine records within the same cluster, significantly reducing the overall computational burden while ensuring complete duplicate removal.
3Productivity
If cluster size is increased to reduce the number of processing steps, then processing overhead is reduced, but memory requirements and computational complexity per cluster increase
Solution Approach 1:
The patent creates multiple smaller clusters rather than one large cluster, distributing the data processing load across several manageable units. Each cluster contains a controlled number of records that can be processed efficiently in memory, preventing excessive memory consumption while maintaining high processing throughput through parallel or sequential cluster processing.
Solution Approach 2:
The system dynamically adjusts cluster processing based on available resources and data characteristics. By making clusters processable units that can be handled in memory, the system optimizes the balance between processing throughput and memory consumption, allowing efficient duplicate identification without overwhelming system resources.
Data Source
AI summary
Some embodiments of the present invention include a method for identifying duplicate records from a group of records in a database system. The method includes generating a cluster of records from a group of records based on one or more keys; splitting the cluster of records into multiple subsets of records with each subset of records having fewer number of records than the cluster of records, wherein the splitting the cluster of records into multiple subsets of records is based on a number of records in the cluster of records exceeding a threshold; causing duplicate sets of records in each of the subsets of records to be identified, wherein a duplicate set of records includes one or more records, and wherein when a duplicate set of records includes two or more records, the two or more records are duplicates of one another; merging all of the duplicate sets of records identified from the multiple subsets of records forming a first group of duplicate sets of records; and forming a representative set of records based on selecting a representative record from each of the duplicate sets in the first group of duplicate sets of records.