Delayed Propagation for Sliding-Window Aggregations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Processing out-of-order data streams in real-time applications with sliding window aggregations is computationally expensive and leads to bottlenecks due to the need for buffering and reordering, which increases latency and memory/computational overhead, while existing solutions are limited to Boolean values and restricted data types.
Innovation Solution
A method that delays value propagation in a hierarchical graph-based data structure, allowing for partial aggregations to be reused across multiple time windows, supporting arbitrary data types and reducing recomputations by delaying and collecting propagations for efficient computation of aggregations over out-of-order data streams.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If buffering and reordering are used to handle out-of-order stream elements, then the analysis results can be computed correctly, but the latency of the output increases and memory/computational overhead becomes huge
Solution Approach 1:
The patent segments the stream processing into multiple independent threads, each handling a specific time window. This allows parallel processing of different time windows without requiring global buffering and reordering, thus reducing latency while maintaining correctness of aggregation results.
Solution Approach 2:
The patent pre-allocates and pre-initializes data structures for multiple time windows in advance. When stream elements arrive, they can be directly inserted into the appropriate pre-prepared window structures without dynamic allocation or reordering, reducing processing latency and memory overhead.
2Reliability
If buffering and reordering are used to handle out-of-order stream elements, then the analysis results can be computed correctly, but the memory and computational overhead increases quickly
Solution Approach 1:
By dividing the processing into segmented threads for different time windows, each thread maintains only the data structures necessary for its specific window, rather than a single large global buffer. This segmentation reduces memory overhead and computational complexity.
Solution Approach 2:
Each thread independently manages its own time window data structures and aggregation state, serving itself without requiring coordination or global resource management. This self-service approach reduces the computational overhead of synchronization and memory management.
3Measurement precision
If state updates are performed for every newly received stream element, then the aggregation remains accurate, but the computational cost becomes a bottleneck in real-time applications with high data velocity
Solution Approach 1:
The patent segments the aggregation computation across multiple threads, each responsible for specific time windows. This allows parallel updates of aggregation states for different windows simultaneously, increasing processing throughput while maintaining accuracy through independent thread execution.
Solution Approach 2:
The patent maintains continuous aggregation computation by having threads constantly process incoming stream elements for their assigned time windows without idle waiting. This continuous processing maximizes productivity while preserving accuracy through uninterrupted aggregation updates.
4Productivity
If stream elements are processed in the order they are received, then processing is simpler and faster, but the aggregation results may be incorrect when elements arrive out of order
Solution Approach 1:
The patent segments processing into threads that each handle specific time windows with predetermined ordering requirements. Within each segment, elements are processed in the correct order for that window, while segments themselves are processed in parallel, maintaining both correctness and high productivity.
Data Source
AI summary
Systems, computer-readable media and methods for aggregating data items from an out-of-order data stream over a sliding window efficiently. The method delays the value aggregation for certain time windows and computes partial aggregations that can be reused for the multiple time windows. Aggregations may have any value type such as Boolean, integer, strings, floating point, vector and map.