Key-Based Data Aggregation Service for Stream Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data stream processing systems face inefficiencies when handling small-sized data bundles, leading to increased overhead and reduced throughput due to constant processing costs.
Innovation Solution
Implementing key-based aggregation to group data associated with the same key into batches, which are processed only when they meet specific batch thresholds or time criteria, thereby reducing unnecessary processing and improving efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is processed individually as small bundles, then processing can be performed immediately, but overhead increases and throughput decreases
Solution Approach 1:
The patent merges multiple small data bundles into larger batches before processing. Data items with the same key are grouped together into batches, which are then processed collectively rather than individually. This combining approach reduces the per-item processing overhead and improves overall throughput by amortizing fixed processing costs across multiple data items.
Solution Approach 2:
The patent performs preliminary grouping and batching of data items before the actual processing occurs. Data is collected and organized into batches based on key matching, and only when batch criteria are met (size threshold or time threshold) does processing begin. This preliminary organization reduces the overhead of individual processing operations.
2Productivity
If data is aggregated into batches before processing, then overhead is reduced and throughput improves, but processing delay increases
Solution Approach 1:
The patent implements dynamic batch creation and processing where batch thresholds and time criteria can be adjusted. The system dynamically determines when to process batches based on configurable parameters, allowing optimization between throughput and latency requirements. This dynamic approach enables the system to adapt to different workload conditions and performance requirements.
Solution Approach 2:
The patent changes the parameters of data bundles by aggregating multiple small items into larger batches. By adjusting batch size thresholds and time criteria, the system transforms the processing parameters to achieve better throughput while managing acceptable delay levels. This parameter adjustment resolves the contradiction by finding optimal values that balance both concerns.
3Ease of operation
If constant processing is performed on all data, then data is handled immediately, but efficiency decreases due to unnecessary processing
Solution Approach 1:
The patent applies partial processing by only processing batches that meet specific criteria (minimum size threshold or time threshold). Not all data is processed immediately or individually - instead, processing occurs selectively on aggregated batches that justify the processing overhead. This partial action approach improves efficiency by avoiding unnecessary processing of small, insignificant data bundles.
Solution Approach 2:
The patent maintains continuous data collection and aggregation while processing occurs periodically when batch criteria are met. This continuous aggregation followed by periodic processing creates a rhythm of useful action that improves efficiency compared to constant individual processing, while ensuring data is handled in a timely manner.
Data Source
AI summary
Techniques are disclosed for aggregating received data from a data stream. Data is received from a particular stream partition to which a device is subscribed, and subsets of the data, from the particular stream partition, are aggregated based on respective keys associated with the subsets of the data. The device determines whether one or more subsets of data, associated with a particular key, meet at least one processing criteria, such as a threshold amount of data, and refraining from processing the aggregated data when the processing criteria is not met. Once additional subset(s) of data associated with the particular key are received, they are aggregated with the one or more subsets of data. When the processing criteria is satisfied, the device processes the aggregated subsets of data associated with the particular key.