Identity Graph Architecture for Deterministic Entity Resolution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing identity resolution systems are inflexible, prone to data corruption, and lack the ability to decouple data ingestion, identity resolution, and profile computation, leading to non-deterministic results and high risk of data loss or misconfiguration.
Innovation Solution
Decoupling data ingestion, identity resolution, and profile computation by using a Raw Identity Graph tier for recording fragments and a Computation tier for applying rules, allowing asynchronous recomputation and flexible algorithm management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing identity resolution systems are used to collect and process entity data from multiple sources, then identity resolution can be performed, but the systems are inflexible and prone to data corruption
Solution Approach 1:
The system is divided into distinct tiers: a data ingestion tier that collects entity data from multiple sources, a Raw Identity Graph tier that stores fragments without processing, and a Computation tier that applies resolution rules. This segmentation isolates data collection from processing, preventing corruption while maintaining flexibility in each layer independently.
Solution Approach 2:
The Raw Identity Graph tier acts as an intermediary layer between data ingestion and identity resolution computation. It stores identity fragments in a neutral, structured format without applying resolution logic, thereby decoupling data collection from processing and eliminating the coupling that causes data loss and misconfiguration in traditional systems.
2Reliability
If data ingestion, identity resolution, and profile computation are coupled in existing systems, then processing can be simplified, but the systems produce non-deterministic results and have high risk of data loss
Solution Approach 1:
The system separates data ingestion, Raw Identity Graph storage, and Computation tier processing into distinct architectural layers. This segmentation ensures that data fragments are captured deterministically in the Raw Identity Graph tier without being affected by subsequent computation variations, enabling reproducible identity resolution results.
Solution Approach 2:
The Raw Identity Graph tier performs preliminary action by capturing and storing all identity fragments from data sources in a standardized structure before any resolution computation occurs. This preliminary data capture ensures that no data is lost and provides a deterministic foundation for subsequent profile computation, regardless of algorithm changes or reprocessing.
3Reliability
If existing systems process entity data through coupled ingestion and resolution, then processing can be straightforward, but the risk of data corruption and misconfiguration is high
Solution Approach 1:
The Raw Identity Graph tier serves as a protective intermediary that stores identity fragments in a neutral, schema-enforced structure between data ingestion and resolution computation. This intermediary layer protects against data corruption by preventing direct modification of raw data during computation, while version control on the Computation tier enables safe algorithm updates without risking data integrity.
Solution Approach 2:
The system implements beforehand cushioning through version control mechanisms on the Computation tier and validation rules in the Raw Identity Graph tier. These measures are established in advance to prevent data corruption from algorithm errors or misconfigurations, allowing the system to withstand computational failures without losing the integrity of the captured identity fragments.
4Adaptability or versatility
If identity resolution systems use unified processing for data ingestion and profile computation, then the system can be simpler to operate, but it lacks flexibility for parallel engineering efforts and algorithm management
Solution Approach 1:
The architecture segments the system into independent tiers with clearly defined interfaces: the Data Ingestion tier feeds the Raw Identity Graph tier, which in turn feeds the Computation tier. This segmentation allows different teams to work on each tier independently with version control, enabling parallel engineering efforts while maintaining operational simplicity through standardized data flow between tiers.
Solution Approach 2:
The Computation tier is designed to be dynamic, allowing different identity resolution algorithms to be applied to the same Raw Identity Graph data. Version control on computation algorithms enables flexible algorithm management and A/B testing without affecting data ingestion or storage, providing adaptability while maintaining operational consistency through the stable Raw Identity Graph interface.
Data Source
AI summary
A method of resolving an identity of an entity is disclosed. Entity data is ingested. Identity fragments observed in the entity data are recorded. All or a subset of identifiers that are connected across the identity fragments are discovered. A directed graph of the identity fragments is built. An entity profile is produced by applying a set of rules to the directed graph.