Adaptive Match Key Generation for Customer Record Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Companies face significant challenges in managing customer data due to frequent changes, inconsistencies, and invalid information across millions of records, leading to intensive system resource consumption in traditional database systems used for customer data management.
Innovation Solution
The generation of adaptive match keys through a system that creates a graph of nodes and edges representing customer data attributes, allowing for efficient matching and clustering of records, reducing resource consumption while achieving near-perfect recall.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional intensive multiple-value matching process is applied to all records, then matching accuracy is improved, but system resource consumption increases significantly
Solution Approach 1:
The patent segments the matching process into two distinct stages: shallow matching that uses index lookups to quickly narrow down candidate records, and deep matching that applies intensive multiple-value comparison only to the small subset of candidate records. This segmentation allows the system to maintain high matching accuracy while dramatically reducing overall resource consumption by avoiding exhaustive comparison of all record pairs.
Solution Approach 2:
The patent performs preliminary shallow matching using index lookups on selected fields before conducting the intensive deep matching process. This preliminary action filters out non-candidate records early, so that the resource-intensive multiple-value matching is applied only to records that have already been pre-identified as potential matches, thereby reducing total resource consumption while preserving accuracy.
2Speed
If shallow matching with index lookups is used, then processing speed is improved, but matching completeness decreases
Solution Approach 1:
The patent introduces an intermediary deep matching process that bridges the gap between shallow index-based matching and exhaustive matching. The shallow matching serves as a fast filter to identify candidate records, while the subsequent deep matching acts as an intermediary step that performs comprehensive multiple-value comparison on these candidates, ensuring that no true matches are lost while maintaining overall processing efficiency.
Solution Approach 2:
The system performs preliminary shallow matching to quickly identify candidate records, then applies deep matching as a follow-up action to ensure completeness. This two-stage approach with preliminary filtering followed by comprehensive verification maintains matching completeness by ensuring that all candidate records undergo thorough comparison, while still benefiting from the speed of the initial filter.
3Measurement precision
If match keys are generated from all possible record combinations, then matching accuracy is improved, but device complexity increases
Solution Approach 1:
The patent applies local quality by generating match keys selectively based on the specific characteristics and requirements of different data sources and matching scenarios. Rather than using a uniform approach for all records, the system determines appropriate match key generation strategies locally for each matching context, considering factors such as data quality, source reliability, and expected match rates, thereby maintaining accuracy while managing complexity.
Solution Approach 2:
The patent implements dynamic match key generation where the selection and composition of match keys adapts based on the matching context, data characteristics, and performance requirements. The system can dynamically adjust which fields are used for match key generation, how many match keys are created per record, and the stringency of matching criteria, allowing it to optimize between accuracy and complexity for different operational scenarios.
Data Source
AI summary
A system creates graph of nodes connected by edges. Each node represents corresponding value of corresponding attribute and is associated with count of corresponding value. Each edge is associated with count of instances that values represented by corresponding connected nodes are associated with each other. The system identifies each node associated with first count as first set of keys, and deletes each node associated with first count. The system identifies each edge associated with second count as second set of keys, and deletes each edge associated with second count. The system identifies each node associated with third count as third set of keys, and deletes each node associated with third count. The system identifies each edge associated with fourth count as fourth set of keys, and deletes each edge associated with fourth count. The system uses each set of keys to search and match records.