Sliding Window Stream Processing with Incremental Caching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional data processing systems, especially those handling big data streams, face challenges in scalability, fault tolerance, and resource management due to fixed computational resources, leading to inefficiencies in processing high volumes of data in real-time with varying streaming rates.
Innovation Solution
A method and system that utilize a sliding window approach to process data streams efficiently by selecting consecutive events, applying functions incrementally, and optimizing resource usage through caching relevant events in memory and storing others in non-volatile storage, allowing for low-latency computation and adaptable resource allocation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional batch processing systems are used to process data streams, then data can be accessed multiple times for processing, but the system cannot handle continuous data arrival in real-time and requires storing data for future reference
Solution Approach 1:
The patent segments the stream processing system into multiple independent operators that can be distributed across different computing nodes. Each operator processes a portion of the data stream independently, enabling parallel processing that achieves real-time speed while the distributed architecture provides fault tolerance through redundancy and checkpointing capabilities.
2Productivity
If window stream operators store all events within the filter scope in local memory, then processing is fast, but the system cannot handle large volumes of events that exceed memory capacity
Solution Approach 1:
The patent transitions from a single-dimension memory storage approach to a multi-dimensional storage hierarchy. Events are organized into time-based windows and stored across multiple dimensions including memory, disk storage, and distributed storage systems. This dimensional expansion allows the system to handle large volumes of events by distributing them across different storage layers rather than being constrained by single memory capacity limits.
3Adaptability or versatility
If fixed computational resources are allocated to stream processing systems, then resource management is simple, but the system cannot adapt to changing streaming data rates resulting in over-provisioning or under-provisioning
Solution Approach 1:
The patent implements dynamic resource allocation where computational resources are automatically adjusted based on the current data stream characteristics. The system monitors data arrival rates, window sizes, and processing backlogs, then dynamically scales operator instances and allocates computing resources accordingly. This dynamic adaptation eliminates both over-provisioning and under-provisioning while maintaining simple resource management through automated control loops.
4Measurement precision
If a sliding window selects a large number of consecutive events for processing, then processing accuracy is improved, but the computation time and resource requirements increase
Solution Approach 1:
The patent applies preliminary actions by pre-computing and caching aggregation results for smaller event subsets within the sliding window. As events arrive continuously, the system maintains incremental aggregation states and pre-calculates intermediate results that can be quickly combined to produce final window aggregations. This preliminary computation approach maintains high processing accuracy for large windows while significantly reducing computation time by avoiding recalculation of all events from scratch.
Data Source
AI summary
An input stream of events is processed to obtain an output stream of events. Consecutive events are selected from the input stream using a sliding window to obtain sliding window events, then a function is applied thereto to obtain an output result value. Operations of: outputting the output result value in the output stream; splitting the sliding window events into filter-complying events and pending events; applying the function on the pending events to obtain preliminary value(s); selecting, from the input stream, a second plurality of events; adding the second plurality of events to the sliding window events; removing, from the sliding window events, the filter-complying events to obtain a new set of sliding window events; and applying the function to the second plurality of events and the preliminary value(s) to obtain a new output result value, are then iteratively performed.