Stream Join Memory Optimization via Time-Bound Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In stream processing, large memory and storage resources are consumed when joining multiple data streams due to the need to hold data from one stream in memory or storage while waiting for matching events from another stream, leading to system overload and inefficiency.
Innovation Solution
Analyzing join conditions to determine allowable time ranges for data matching, allowing for intentional delays in ingesting later-arriving stream data and employing multi-stage join plans with increasing delays to reduce memory and storage requirements without increasing latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data from one data stream is held in memory or storage while waiting for matching events from another data stream, then join accuracy is improved, but memory and storage consumption increases significantly
Solution Approach 1:
The patent applies preliminary action by analyzing join conditions beforehand to determine time ranges and processing orders. The system pre-calculates which data stream should be pulled first and the time ranges for allowable joins, eliminating the need to hold data in memory while maintaining join accuracy.
Solution Approach 2:
The patent implements dynamics by making the data processing approach adaptive based on analyzed join conditions. The system dynamically determines processing orders and time ranges for different join scenarios, rather than using a fixed memory-holding approach for all cases.
2Device complexity
If a fixed time window is used for join operations, then system simplicity is improved, but adaptability to changing correlations deteriorates
Solution Approach 1:
The patent applies parameter changes by making the time window dynamic rather than fixed. The system analyzes join conditions to determine appropriate time ranges for each specific join operation, adapting the time parameter based on the characteristics of the data streams being joined.
3Adaptability or versatility
If large window sizes are used to accommodate dynamic correlations, then adaptability is improved, but system overload increases
Solution Approach 1:
The patent applies local quality by making the time window size specific to each join condition rather than using a uniformly large window for all operations. The system determines appropriate time ranges based on the local characteristics of each join, using only the necessary window size for each specific case.
Solution Approach 2:
The patent applies partial action by using only the necessary portion of the time window required for each join operation. Rather than maintaining large windows for all possible scenarios, the system calculates and uses minimal adequate time ranges based on actual join conditions.
4Reliability
If data is held in memory for the match interval, then join completeness is improved, but resource efficiency deteriorates
Solution Approach 1:
The patent applies preliminary action by pre-analyzing join conditions to determine processing orders and time ranges. This eliminates the need to hold data in memory during processing, as the system knows in advance which data to pull and when, maintaining join completeness while improving resource efficiency.
Data Source
Figure 1
Figure 2a
Figure 2b
AI summary
Large amounts of memory can be consumed in streaming joins because events from one stream are held in memory while waiting for matching events from a second stream. Memory needs can be reduced by analyzing the join condition to determine the bounds on the time discrepancy between events in the two streams. When it is determined that an event from one stream must occur prior to the matching event from the other stream, the later-arriving stream data can be ingested with an intentional delay. When it is determined that regardless of input received from a first stream, no output will be produced when there is no input from the second stream, pulling data from the first stream can cease. A multi-stage join plan can be employed so that a less busy stream can be scanned with increasing amounts of intentional delay. Only unmatched data is stored.