Persistent Entity Index for Incremental Data Conflation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In the online search environment, determining whether entity-attribute pairs from different content feeds refer to the same entity is challenging due to varying schemas and differing values, complicating the identification of related sources and accuracy.
Innovation Solution
A persistent entity index is used to match and aggregate entity-attribute pairs, allowing incremental updates and reconciliation of information, enabling efficient matching of additional data without re-matching existing information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If entity-attribute pairs from multiple content feeds are matched and reconciled to ensure accuracy, then information accuracy is improved, but processing time and computational resources increase
Solution Approach 1:
The system performs preliminary actions by pre-processing content feeds to extract and normalize entity-attribute pairs before the matching phase. Entities are pre-identified and stored in a structured format, allowing the matching algorithm to work with cleaned data rather than raw content, thereby improving accuracy while reducing processing time during the actual conflation operation.
Solution Approach 2:
The matching process is segmented into multiple stages: exact matching based on entity identifiers, fuzzy matching for similar names, and attribute-based matching for verification. This segmentation allows the system to quickly eliminate non-matches through exact matching before applying more computationally intensive fuzzy matching techniques, thus maintaining accuracy while optimizing processing time.
2Reliability
If all entity information is re-matched each time new information is received, then information freshness is improved, but computational resources and processing time increase significantly
Solution Approach 1:
The system performs preliminary actions by maintaining a persistent entity index that is pre-built from existing content feeds. When new information arrives, the system only needs to match against this pre-organized index rather than re-processing all historical data, ensuring information freshness while preserving processing efficiency through incremental updates.
Solution Approach 2:
The entity index is designed to be dynamic and incremental, allowing the system to add new entities and update existing ones without re-matching the entire dataset. The index structure supports efficient insertion and update operations, enabling the system to maintain current information while avoiding the computational overhead of full re-matching.
3Measurement precision
If entity matching uses detailed attribute comparison to handle varying schemas, then matching accuracy is improved, but device complexity increases
Solution Approach 1:
The system applies local quality by using different matching strategies for different attributes based on their reliability and variability. Core identifiers like entity IDs or unique names receive exact matching treatment, while optional attributes with varying schemas use fuzzy matching or normalization-based comparison. This localized approach to attribute matching improves overall accuracy without requiring complex handling of every possible attribute type.
Solution Approach 2:
The matching system employs universal normalization techniques that can handle multiple attribute types and schemas through a common framework. By converting diverse attribute formats into a standardized representation, the system achieves high matching accuracy across varying schemas while avoiding the need for separate complex processing logic for each attribute type, thus reducing overall system complexity.
Data Source
AI summary
Systems, methods, and computer-readable storage media are provided for conflating entities using a persistent entity index. Information (including attributes) pertaining to a plurality of entities is received. The received information is either matched with one or more existing entities in the persistent entity index or, if no match is found, selected for addition to the persistent entity index. The persistent entity index includes entity-attribute pairs associated therewith. Attributes associated with matching entities for which information is received are aggregated and/or reconciled with the entity-attribute pairs associated with existing entities included in the persistent entity index. The persistent entity index may be incrementally updated at predetermined time intervals to insure the accuracy and freshness of the information associated therewith.