Dynamic Operator Fusion for Distributed Stream Computing Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed computing environments, software updates can negatively impact stream computing systems by requiring temporary suspension of non-updated jobs and causing bottlenecks, as processing elements and operators are often restarted during updates, leading to unnecessary downtime.
Innovation Solution
The system intelligently fuses and unfuses operators among processing elements, allowing updates to occur at runtime without suspending non-updated jobs by redirecting operators to unaffected processing elements, thus minimizing downtime and avoiding bottlenecks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If processing elements are restarted during software updates, then the software update can be applied, but the stream computing job experiences downtime and reduced productivity
Solution Approach 1:
The system segments the processing element into multiple operators that can be independently managed. During a software update, only the affected operators are unfused and updated, while other operators remain fused and continue executing their functions. This segmentation allows partial updates without complete system restart, maintaining job execution continuity while applying software updates.
2Adaptability or versatility
If processing elements are restarted during software updates, then the updated software version can be deployed, but unnecessary downtime and bottlenecks occur
Solution Approach 1:
The system performs preliminary identification of affected operators before executing the software update. By determining which operators will be impacted by the update in advance, the system can selectively unfuse only those specific operators for updating while keeping other operators fused and running. This preliminary action prevents unnecessary restarts and minimizes downtime during software version deployment.
3Productivity
If all operators are fused to a processing element, then resource utilization is optimized, but software updates require complete processing element restart
Solution Approach 1:
The system dynamically manages the fusion state of operators with processing elements based on update requirements. Operators can be transitioned between fused and unfused states as needed. During normal operation, operators remain fused for optimal resource utilization. When software updates are required, the system dynamically unfuses only the affected operators, applies updates, and then refuses them, providing update flexibility while maintaining resource efficiency.
Data Source
AI summary
A computer system configures processing elements within a distributed computing system. A processing element within a distributed computing environment is determined to be affected by a software update, wherein each processing element of the distributed computing system includes a plurality of components and the software update modifies the components of the processing elements. The determined processing element is split into a plurality of processing elements based on a set of factors. The plurality of components of the processing element are assigned among the plurality of processing elements based on components affected by the software update. Embodiments of the present invention further include a method and program product for configuring processing elements within a distributed computing system in substantially the same manner described above.