Event-Driven Identity Graph Conflation in Distributed Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in efficiently conflating disjoint identity graph subgraphs across multiple processing nodes in a distributed system, leading to increased latency and processing bottlenecks.
Innovation Solution
The implementation of an event-driven identity graph conflation method using a Kafka Streams application and Kafka cluster, which processes conflation events asynchronously, allowing for real-time and eventually consistent merging of subgraphs without blocking ongoing processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional batch processing is used to conflate identity graph subgraphs, then data consistency is ensured, but processing latency increases and system throughput decreases
Solution Approach 1:
The system transitions from static batch processing to dynamic event-driven processing. Conflation operations are triggered dynamically by events (e.g., new user data, graph updates) rather than executing periodically. This allows the system to maintain data consistency while reducing latency by processing only when necessary, rather than waiting for batch intervals.
Solution Approach 2:
The system performs preliminary actions by maintaining subgraphs in a ready-state with pre-computed data structures. When conflation events occur, the pre-prepared subgraphs can be merged immediately without waiting for data collection or transformation, significantly reducing processing latency while ensuring consistency through event-triggered validation.
2Measurement precision
If synchronous conflation processing is implemented, then data accuracy is maintained, but system throughput and concurrent processing capability deteriorate
Solution Approach 1:
The conflation process is segmented into independent, parallelizable tasks. Each conflation event is treated as an independent unit of work that can be processed concurrently with other events. The system divides the global conflation problem into local subgraph conflations that can execute in parallel, maintaining data accuracy through event isolation while dramatically improving throughput.
Solution Approach 2:
An event-driven intermediary layer (Kafka event stream) is introduced between data sources and conflation processing. This intermediary buffers and routes conflation events asynchronously, allowing multiple concurrent processing streams without blocking. The intermediary ensures data accuracy through structured event schemas while enabling high-throughput parallel processing.
3Adaptability or versatility
If distributed subgraph storage is used across multiple nodes, then system scalability is improved, but conflation processing complexity and coordination overhead increase
Solution Approach 1:
Each distributed node autonomously handles conflation events for its local subgraphs without requiring centralized coordination. Nodes independently process conflation events, manage their own data structures, and maintain consistency through self-contained event processing. This self-service approach eliminates complex inter-node coordination while preserving distributed scalability.
Solution Approach 2:
The system uses homogeneous data structures and processing logic across all distributed nodes. Each node maintains subgraphs with the same schema and processes conflation events using identical algorithms. This homogeneity simplifies distributed processing by eliminating the need for complex data transformation or coordination between heterogeneous nodes, reducing processing complexity while maintaining scalability.
Data Source
AI summary
A distributed system including multiple processing nodes. The distributed system can perform certain acts. The acts can include receiving a first conflation event identifying a first node and a second node. The first node can be part of a first set. The first set can include a sole parent node stored at a first processing node of the multiple processing nodes. The second node can be part of a second set. The second set can include a sole parent node stored at a second processing node of the multiple processing nodes. The first and second sets can be disjoint sets. The first conflation event can be received at an event-driven stream application at one of the multiple processing nodes. The acts also can include conflating the first set and the second set into a conflated set. The conflated set can include the first and second nodes. The conflated set can include a sole parent node. Data for each node of the conflated set can be stored at the sole parent node of the conflated set at a processing node of the multiple processing nodes. Other embodiments are disclosed.