Two-Level Cache for Entity Resolution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current solutions for active learning of entity resolution (ER) do not scale well with large data sets, requiring several minutes to tens of minutes per iteration on a 6-node cluster, and lack efficient methods to reduce redundant computations.
Innovation Solution
Implementing a distributed compute memoing cache with a two-level cache hierarchy to manage link feature vector tables, allowing for pre-computation or on-the-fly computation of intermediate results, and optimizing caching based on frequency, storage, and processing costs to eliminate redundant computations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If active learning is performed on large scale entity resolution data sets, then entity resolution accuracy is improved, but computation time increases significantly (several to tens of minutes per iteration on a 6-node cluster)
Solution Approach 1:
The patent pre-computes and caches link feature vectors for all possible entity pairs before the active learning process begins. This preliminary computation stores intermediate results that can be reused across multiple active learning iterations, eliminating the need to recompute the same feature vectors repeatedly and significantly reducing iteration time.
Solution Approach 2:
The patent creates and maintains cached copies of link feature vector tables that can be quickly retrieved and reused. Instead of computing feature vectors from scratch during each active learning iteration, the system copies and reuses pre-computed feature vectors from the cache, dramatically speeding up the entity resolution process while maintaining accuracy.
2Measurement precision
If active learning iterations are performed on large data sets, then entity resolution rules are refined, but scalability deteriorates (current solutions do not scale well)
Solution Approach 1:
The patent segments the computation process into distinct phases: pre-computation of link feature vectors, caching of intermediate results, and execution of active learning iterations. This segmentation allows the system to handle large data sets more efficiently by preparing computational resources in advance and reusing them across iterations, improving scalability.
Solution Approach 2:
The patent implements a caching mechanism that creates reusable copies of link feature vector tables. These cached copies enable the system to scale to larger data sets by avoiding redundant computations, allowing active learning iterations to proceed faster even as data set size increases.
3Power
If blocking functions are used to reduce the number of pairs to be compared, then computation cost is reduced, but the number of pairs still remains large for large scale data sets
Solution Approach 1:
The patent performs preliminary computation of link feature vectors for all pairs that pass blocking functions, storing these intermediate results in a cache. This pre-computation approach reduces the computational burden during active learning iterations by having already processed the blocking and feature extraction, even though the number of pairs remains large.
Solution Approach 2:
The patent creates cached copies of link feature vectors for blocked pairs, allowing rapid reuse of these computations across multiple active learning iterations. This copying mechanism reduces the effective computation cost by avoiding redundant processing of the same pairs throughout the iterative process.
Data Source
AI summary
One embodiment provides for a method that includes performing, by a processor, active learning of large scale entity resolution using a distributed compute memoing cache to eliminate redundant computation. Link feature vector tables are determined for intermediate results of the active learning of large scale entity resolution. The link feature vector tables are managed by a two-level cache hierarchy.