Graph-Based Entity Resolution for Organization Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Entity resolution in databases is challenging due to the presence of duplicates from multiple sources with varying capitalization and abbreviation conventions, leading to decreased data quality and difficulty in answering complex queries.
Innovation Solution
A machine learning pipeline with graph-based blocking and clustering strategies, utilizing MapReduce architecture and Hadoop implementation, to normalize and group records by shared properties, assign probability scores, and cluster records into unique entities, addressing noise and inconsistencies in organization names and relations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional entity resolution methods are used to identify duplicate records, then data quality can be improved, but the computational complexity and processing time increase significantly with large databases
Solution Approach 1:
The patent divides the entity resolution process into two distinct phases: blocking (segmenting records into smaller groups based on shared properties) and clustering (resolving duplicates within each block). This segmentation reduces the computational complexity from O(n²) comparisons to O(n) by processing records in manageable chunks rather than comparing all pairs simultaneously.
Solution Approach 2:
The blocking phase performs preliminary grouping of records based on shared properties (organization names, addresses, phone numbers) before the actual duplicate resolution occurs. This preliminary action organizes data in advance so that the clustering phase only needs to compare records within the same block, significantly reducing processing time while maintaining resolution accuracy.
2Measurement precision
If all record pairs are compared to ensure accurate entity resolution, then measurement precision improves, but the quantity of computations becomes intractable for massive databases
Solution Approach 1:
The patent segments the comparison task by first grouping records into blocks based on shared properties. Within each block, records are more likely to represent the same entity, so comparisons are concentrated where they are most needed. This reduces the total number of comparisons from n(n-1)/2 to a much smaller number while maintaining high resolution accuracy through the clustering phase.
Solution Approach 2:
The patent applies different processing quality to different regions of the data: records within the same block receive intensive comparison and clustering analysis, while records in different blocks are assumed to be distinct entities. This local quality approach concentrates computational resources on areas most likely to contain duplicates, achieving high precision without exhaustive comparison.
3Measurement precision
If normalization and clustering strategies are applied to handle varying capitalization and abbreviation conventions, then entity resolution accuracy improves, but device complexity increases
Solution Approach 1:
The patent segments the data processing into distinct stages: normalization (handling capitalization and abbreviations), blocking (grouping by shared properties), and clustering (resolving duplicates). Each stage handles a specific aspect of the problem, making the overall complex task manageable through modular processing steps that can be implemented and maintained separately.
Solution Approach 2:
The patent introduces intermediate structures (blocks and clusters) as mediators between the raw input data and the final resolved entities. These intermediaries organize data in ways that facilitate accurate matching while handling variations in capitalization and abbreviations, acting as buffer layers that simplify the resolution process.
Data Source
AI summary
A novel entity resolution approach for the organization entity domain can be implemented in the MapReduce framework with low memory requirements so that it may scale to large scale datasets. A new clustering approach, sClust, significantly improves the recall of the pairwise classifier.