Parallel Data Stream Processing via Common Partitioning Keys
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data stream processing technologies face challenges in efficiently processing large volumes of data in real-time due to increased communication traffic and limited plan derivation for parallel distributed processing, leading to high network resource consumption.
Innovation Solution
A method that extracts relationships between properties in data streams to specify common partitioning keys for grouping queries, assigning these queries to parallel nodes for distributed processing, thereby reducing inter-node communications and optimizing data processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data streams are processed sequentially using traditional stream processing systems, then processing simplicity is maintained, but processing speed and real-time capability deteriorate when handling large volumes of data
Solution Approach 1:
The patent divides the data stream processing system into multiple independent parallel processing nodes, each capable of processing specific portions of the data stream simultaneously. This segmentation enables scalable horizontal expansion to handle large volumes of data while maintaining processing simplicity at the node level, resolving the contradiction between processing speed and system complexity.
2Productivity
If parallel distributed processing is implemented without query grouping, then processing capacity increases, but network resource consumption increases due to excessive inter-node communications
Solution Approach 1:
The patent merges queries that share common partitioning keys into partitioning groups, which are then assigned to the same processing node. This combining strategy reduces the number of inter-node data transmissions by keeping related processing operations co-located, thereby maintaining high processing capacity while significantly reducing network resource consumption.
3Productivity
If queries are assigned to multiple parallel nodes for distributed processing, then processing throughput increases, but communication overhead increases due to inter-node data exchange
Solution Approach 1:
The patent performs preliminary grouping of queries by common partitioning keys before assigning them to parallel nodes. This advance organization ensures that queries within each partitioning group are co-located on the same node, eliminating the need for complex runtime coordination and data exchange between nodes, thus achieving high throughput with minimal communication overhead.
Data Source
AI summary
A non-transitory computer-readable storage medium stores a data stream processing program that causes a computer to execute a process. The process includes extracting a relationship between properties included in events in a data stream from the data stream itself, definitions of queries for processing the data stream, or both of the data stream and the definitions of the queries; specifying a common partitioning key common to a subset of the queries to be grouped into a partitioning group based on partitioning keys in the definitions of the queries, the relationship between the properties, and a probability of the relationship; and assigning the subset of the queries belonging to the partitioning group to each of parallel nodes provided for parallel distributed processing based on the specified common partitioning key.