Runtime Grouping of Streaming Processing Elements
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In streaming computing, conventional resource managers face challenges in allocating resources efficiently due to constant usage and lack of knowledge about stream processing topology, leading to potential breaks in processing graphs and difficulties in adding or removing machines without introducing latency.
Innovation Solution
A method that dynamically groups and un-groups processing operators and elements, monitoring resource utilization to identify candidates for parallelization and fusion, adjusting these configurations at runtime to optimize resource allocation without user intervention, using techniques like graph transforms and tuple flow rate monitoring.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional resource managers are used to allocate machines in streaming computing, then resource allocation can be performed, but the system cannot safely remove or add machines without introducing breaks in the processing graph or latency
Solution Approach 1:
The system dynamically adjusts the grouping and un-grouping of processing operators and elements at runtime based on monitored resource utilization metrics. This allows the processing graph to adapt to changing resource availability while maintaining continuity, as the system can reconfigure processing elements without breaking the graph by using dynamic threshold-based decisions for when to group or un-group operators.
Solution Approach 2:
The system continuously monitors distributed processing elements resource utilization and uses this feedback to identify candidate operators and processing elements for parallelization or fusion. This feedback loop enables the system to make informed decisions about resource allocation and reconfiguration, ensuring that changes are made only when resource utilization thresholds are exceeded, thereby maintaining processing graph integrity.
2Productivity
If processing elements are grouped to improve resource utilization, then efficiency increases, but the system becomes less flexible in adapting to changing resource availability
Solution Approach 1:
The system implements dynamic grouping and un-grouping of processing elements based on real-time resource utilization monitoring. When resource utilization exceeds an upper threshold, the system identifies candidates for un-grouping to improve flexibility. When utilization falls below a lower threshold, the system groups candidates to improve efficiency. This dynamic approach allows the system to optimize both productivity and adaptability based on current resource conditions.
Solution Approach 2:
The system changes the operational parameters of processing elements by dynamically adjusting their grouping state based on resource utilization thresholds. This parameter change enables the system to transition between different operational modes (grouped for efficiency, un-grouped for flexibility) without requiring fundamental architectural changes, thus resolving the contradiction between productivity and adaptability.
3Productivity
If the system monitors and dynamically adjusts processing elements at runtime, then resource allocation is optimized, but system complexity increases
Solution Approach 1:
The system performs self-service by automatically monitoring its own resource utilization and making decisions about grouping and un-grouping processing elements without external intervention. The resource manager autonomously identifies candidate operators and processing elements, evaluates them against thresholds, and executes parallelization or fusion operations, thereby optimizing resource allocation while managing complexity through automation rather than manual control.
Solution Approach 2:
The system introduces a resource manager as an intermediary layer between the processing elements and the external environment. This intermediary monitors resource utilization, makes decisions about grouping and un-grouping, and manages the complexity of runtime adjustments. By centralizing control in this intermediary, the system can optimize resource allocation while isolating the complexity of dynamic management from the individual processing elements.
Data Source
AI summary
A method, computer program product, and computer system for dynamically grouping and un-grouping processing operators and processing elements used by a streaming application. A distributed processing elements utilization of resources may be monitored to identify candidate operators and candidate processing elements for at least one of parallelization and fusion. At runtime, via at least one of parallelization and fusion, the grouping and un-grouping of the identified candidate operators and candidate processing elements may be dynamically adjusted.