Stateful Joining of Un-synchronized Data Streams
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in merging and synchronizing continuous and un-synchronized data streams from various sources in real-time, leading to high latency and difficulties in detecting security threats and user behavior anomalies.
Innovation Solution
A scalable and fault-tolerant system that uses processors to create objects by merging data from multiple streams, routing events to retry streams, and transmitting data when synchronization thresholds are met, ensuring low latency and high data quality for downstream applications like machine learning models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data from multiple un-synchronized streams is merged in real-time, then detection speed of security threats improves, but data consistency and synchronization quality deteriorate
Solution Approach 1:
The system performs preliminary actions by pre-processing and buffering data from multiple streams before merging. Events are buffered in-memory with timestamps, and the system pre-establishes joining logic based on time windows and event types, enabling fast real-time merging without compromising consistency
Solution Approach 2:
The system dynamically adjusts merging parameters such as time window sizes, buffering thresholds, and joining strategies based on stream synchronization status and data arrival patterns. This dynamic adaptation allows the system to maintain data consistency while optimizing for detection speed under varying conditions
2Manufacturing precision
If batch processing is used to ensure data consistency, then data quality improves, but latency increases making real-time detection impossible
Solution Approach 1:
The system segments the data processing pipeline into distinct streaming and batch processing paths. Critical security events are processed immediately through the streaming path with low latency, while less time-sensitive data is handled by batch processing to ensure high data quality. This segmentation allows simultaneous optimization of both speed and accuracy for different event types
Solution Approach 2:
The system maintains continuous data flow processing through in-memory buffering and incremental joining operations. Instead of waiting for complete batch data, the system continuously merges events as they arrive within defined time windows, ensuring uninterrupted real-time detection while maintaining data quality through continuous validation and synchronization checks
3Measurement precision
If complex entity resolution is performed across multiple streams, then detection accuracy of anomalies improves, but processing complexity and computational resources increase
Solution Approach 1:
The system performs preliminary entity resolution by pre-computing entity identifiers and attributes from incoming streams before anomaly detection. Entities are resolved and normalized in advance using common identifiers, creating a pre-processed entity model that reduces the complexity of subsequent anomaly detection operations while maintaining high detection accuracy
Solution Approach 2:
The system introduces an intermediary entity resolution layer that acts as a mediator between raw data streams and anomaly detection algorithms. This intermediary layer standardizes entity representations, resolves ambiguities, and enriches events with entity context, thereby improving detection accuracy without significantly increasing the complexity of the core detection logic
Data Source
AI summary
The present disclosure is directed to a scalable, extensible, fault-tolerant system for stateful joining of two or more streams that are not fully synchronized, event ordering is not guaranteed, and certain events arrive a bit late. The system can ensure to combine the events or link the data in near real-time with low latency to mitigate impacts on downstream applications, such as ML models for determining suspicious behavior. Apart from combining events, the system can ensure to propagate the needed entities to other product streams or help in entity resolution. If any of the needed data is yet to arrive, a user can configure a few parameters to achieve desired eventual and attribute consistency. The architecture is designed to be agnostic of stream processing framework and can work well with both streaming and batch paths.