Custom Partitioning of Streaming Data to Reduce Thread Contention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managing large volumes of streaming data from diverse sources with varying attributes is burdensome due to the high speed and volume of information, requiring effective partitioning solutions to handle and process efficiently.
Innovation Solution
A customizable mechanism for partitioning streaming data within an event processing framework, allowing users to configure data streams based on custom attributes such as IP addresses or geographic locations, using object-oriented classes or Java contracts to process sub-streams in parallel and generate sub-streams through hashing functions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data streams are processed at high speed with large volume, then information processing capability is improved, but management complexity and burden increase
Solution Approach 1:
The patent applies segmentation by dividing a single data stream into multiple sub-streams based on custom attributes (e.g., IP address ranges, geographic locations). Each sub-stream is processed independently by different query instances, distributing the processing load and reducing management complexity while maintaining high processing capability.
Solution Approach 2:
The patent introduces a new dimension of customization by allowing users to define custom partitioning attributes and logic beyond traditional fixed partitioning. This enables flexible, attribute-based routing that adapts to specific processing needs without increasing overall system complexity.
2Productivity
If custom partitioning is implemented, then processing efficiency and throughput are improved, but system configuration complexity increases
Solution Approach 1:
The patent implements dynamic partitioning where the system can adapt partitioning logic based on runtime conditions and custom attributes. The partitioning configuration is not static but can be modified through user-defined code, allowing the system to optimize throughput for different scenarios without requiring complex manual configuration.
Solution Approach 2:
The system enables self-service configuration through user-provided code that defines custom partitioning logic. Users can independently configure and modify partitioning behavior without requiring complex system administrator intervention, reducing the burden of system configuration while maintaining high throughput.
3Quantity of substance
If data streams contain multiple attributes from diverse sources, then data comprehensiveness is improved, but processing difficulty increases
Solution Approach 1:
The patent applies local quality by allowing different processing logic to be applied to different attributes and sub-streams. Each sub-stream can be processed with customized logic tailored to its specific attributes, making it easier to handle diverse data sources comprehensively without overwhelming processing difficulty.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Techniques for partitioning data streams are provided. In some examples, a query for processing at least a portion of a data stream may be identified. The data stream may be associated with a user. Additionally, in some examples, code identifying an attribute of the identified stream may be received. The code may be capable of configuring the query based at least in part on the attribute. Further, in some aspects, the code may be configured to partition the data stream into at least a sub-stream based at least in part on the attribute.