In-Memory Event Correlation with Time-Windowed Buffering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Inefficient data processing in large data handling systems, such as digital advertising systems, due to the challenge of correlating incoming events with base events in high-volume data streams, leading to uneconomical and unprofitable operations.
Innovation Solution
Implementing an in-memory state with a limited time window to buffer incoming events for efficient correlation between base and follow-on events, creating aggregation tables for matched pairs and storing uncorrelated events in a log table for less-efficient late querying.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If all incoming events are stored in the in-memory state for correlation, then correlation efficiency is improved, but memory resource consumption increases
Solution Approach 1:
The patent segments the event storage system into two distinct parts: an in-memory state for storing base events within a time window, and a log table for storing events that cannot be correlated. This segmentation allows the system to maintain high correlation efficiency for time-sensitive events while avoiding the memory overhead of storing all events in memory indefinitely.
Solution Approach 2:
The patent applies local quality by treating different types of events with different storage strategies. Base events are stored in the in-memory state with their associated time windows for efficient correlation, while follow-on events that cannot be correlated are redirected to the log table. This localized differentiation optimizes memory usage by only keeping necessary events in memory.
2Quantity of substance
If events are stored in the in-memory state for a limited time window, then memory resource consumption is reduced, but correlation completeness may be compromised
Solution Approach 1:
The patent introduces the log table as an intermediary storage mechanism. When follow-on events arrive outside the time window of base events in the in-memory state, the system checks the log table to see if matching base events exist there. This intermediary log table ensures that correlation completeness is maintained even for events that cannot be correlated within the strict time window constraints.
Solution Approach 2:
The patent performs preliminary actions by first attempting to correlate events within the time window using the in-memory state, and only if that fails does it proceed to check the log table. This preliminary action strategy optimizes performance by quickly handling the majority of correlation cases in memory while having a fallback mechanism for completeness.
3Measurement precision
If database lookups are performed for all event correlation, then correlation accuracy is maintained, but processing speed decreases
Solution Approach 1:
The patent performs preliminary correlation attempts in the in-memory state before resorting to database lookups in the log table. Since most events can be correlated within the time window, the majority of correlation operations are completed using fast in-memory comparisons, maintaining high processing speed while preserving accuracy through the subsequent log table verification for unmatched events.
Data Source
AI summary
In a streaming environment, efficient correlation between base events and relevant follow-on events is accomplished by temporarily storing events in an in-memory state for a limited time window. The in-memory state buffers the incoming stream of events to permit efficient attempted correlation. Successful correlation (e.g., by ID matching) between a follow-on event and a relevant base event, gives rise to specialized aggregation tables for matched (base, follow-on) event pairs. All events are ultimately removed from the in-memory state upon expiry of the time window. This results in correlated matched event pairs being stored in the aggregation tables for efficient querying. Events remaining uncorrelated upon expiration of the time window, are stored only in an (voluminous) log table for less-efficient (but rare) late querying and aggregation. Efficient correlation is achieved by assuming that successful event pair matching is most likely to occur within the in-memory time window, or not at all.